rag-failcase-kit 0.1.0

Lightweight JSONL tracing for debugging failed RAG answers.

rag-failcase-kit

rag-failcase-kit is a lightweight Python package for recording one RAG pipeline run as one JSONL trace.

When a RAG application returns an empty, inaccurate, or weakly relevant answer, it is often hard to tell where the failure happened. The issue may be retrieval quality, prompt construction, fallback logic, answer generation, or a runtime error. In small projects this often becomes a pile of print statements, which is difficult to search, compare, or replay later.

This package keeps the first version intentionally small: it records query, retrieved documents, scores, fallback usage, answer preview, latency, status, and errors into a JSONL file.

Installation

From PyPI:

pip install rag-failcase-kit

From a local checkout:

pip install -e .

For development and release checks:

pip install -e ".[dev]"

30-Second Usage

from rag_failcase_kit import RagLogger

logger = RagLogger(project="demo-rag", log_path="logs/rag_traces.jsonl")

with logger.trace("What is RAG?") as trace:
    trace.log_retrieval(
        [
            {
                "content": "Retrieval-augmented generation combines retrieval with text generation.",
                "source": "docs/rag.md",
                "score": 0.91,
            }
        ],
        top_k=3,
        min_score=0.7,
    )
    trace.log_answer("RAG retrieves relevant context and uses it to generate an answer.")

The log directory is created automatically. Each trace is appended as one JSON object per line.

CLI Usage

The package installs a rag-failcase command:

rag-failcase summary logs/rag_traces.jsonl
rag-failcase failures logs/rag_traces.jsonl
rag-failcase inspect logs/rag_traces.jsonl --trace-id <trace_id>

All CLI commands print JSON output.

Example summary:

{
  "total_traces": 12,
  "successes": 9,
  "failures": 3,
  "fallback_used": 4,
  "avg_latency_ms": 128.42
}

Log Schema

Example JSONL record:

{
  "trace_id": "9fcd7b91-6b12-46fb-8ff1-5f6f1ef49b15",
  "project": "demo-rag",
  "query": "What is RAG?",
  "retrieval": {
    "documents": [
      {
        "content_preview": "Retrieval-augmented generation combines retrieval with text generation.",
        "metadata": {
          "chunk_id": "rag-001"
        },
        "source": "docs/rag.md",
        "score": 0.91
      }
    ],
    "top_k": 3,
    "min_score": 0.7
  },
  "retrieved_count": 1,
  "fallback": {
    "used": false,
    "strategy": null,
    "reason": null
  },
  "answer": "RAG retrieves relevant context and uses it to generate an answer.",
  "answer_preview": "RAG retrieves relevant context and uses it to generate an answer.",
  "latency_ms": 3.241,
  "status": "success",
  "error": null,
  "created_at": "2026-07-09T00:00:00+00:00"
}

Document content is stored as a bounded preview to avoid unexpectedly large logs.

Real RAG Integration

Wrap the part of your RAG pipeline that handles one user query:

from rag_failcase_kit import RagLogger

logger = RagLogger(project="support-bot", log_path="logs/support_rag.jsonl")


def answer_question(query, retriever, llm):
    with logger.trace(query) as trace:
        documents = retriever.search(query)
        trace.log_retrieval(documents, top_k=5, min_score=0.75)

        if not documents:
            trace.log_fallback("no_documents", reason="Retriever returned no documents.")
            answer = "I do not have enough context to answer."
            trace.log_answer(answer)
            return answer

        answer = llm.generate(query=query, documents=documents)
        trace.log_answer(answer)
        return answer

documents can be dictionaries or document-like objects. For objects, the logger safely checks common fields such as page_content, metadata, source, and score.

Optional Integrations

The package does not depend on FastAPI, LangChain, or LangGraph.

FastAPI example:

pip install fastapi uvicorn
uvicorn examples.fastapi_example:app --reload

LangGraph-style example:

python examples/langgraph_example.py

See:

• examples/basic_usage.py
• examples/fastapi_example.py
• examples/langgraph_example.py

Development

pip install -e ".[dev]"
pytest
python -m build
twine check dist/*

To test the console script in a clean environment:

python -m venv .venv-test
. .venv-test/bin/activate
pip install dist/rag_failcase_kit-0.1.0-py3-none-any.whl
rag-failcase --help

Roadmap

• Add prompt logging with explicit redaction controls.
• Add richer failure categorization helpers.
• Add Markdown or HTML reports.
• Add log rotation and sampling controls.
• Add optional adapters for popular RAG frameworks without making them required dependencies.