rag-eval-lite 0.1.4

Structure

Three dataclasses + one evaluator class:

• QueryResult — your input. Pass query_id, retrieved_chunks (ordered list), relevant_chunks (set)
• QueryScore — per-query scores + a failures list with human-readable diagnostics
• EvalReport — mean scores across all queries + failed_queries list
• RAGEvaluator — the main class, just call .evaluate() then .report()

The Failure Diagnostics Logic

Each query gets diagnosed in priority order:

Usage in your package

from rag_eval import RAGEvaluator

dataset = [
    {
        "question": "What is the significance of Apple's manufacturing...",
        "relevent_chunks": ["pdf_chunk_94", "pdf_chunk_17", "pdf_chunk_107"],
        "retrieved_chunks": ["pdf_chunk_94", "pdf_chunk_91", "pdf_chunk_95"]
    },
    ...
]

report = RAGEvaluator.from_dict_list(dataset, k=5)
RAGEvaluator(k=5).report(report)

You can also access report.per_query and report.failed_queries programmatically if you want to log them to MLflow or W&B Weave.