tracixx 0.1.0

AI pipeline observability and debugging toolkit for Databricks

Tracixx — AI Pipeline Observability and Debugging Toolkit

Tracixx is an open-source Python package that gives developers complete observability and debugging capabilities for AI/RAG pipelines built on Databricks.

The Problem

Building RAG (Retrieval-Augmented Generation) pipelines is complex. When they fail, it's hard to know whether the issue came from:

• Data ingestion or chunking
• Embedding quality
• Retrieval scoring
• LLM reasoning

There's no structured way to trace issues across the entire pipeline. Debugging becomes trial-and-error.

The Solution

Tracixx captures every stage of your pipeline:

• Chunks — text, size, source metadata, chunking strategy
• Retrievals — which chunks matched, similarity scores, embedding model
• LLM responses — answer, prompt, context used, token counts

All stored in Delta tables for analysis and debugging. Plus:

• Query replay — re-run a stored query with different configs and compare
• Retrieval eval metrics — Hit Rate, MRR, NDCG@k for labeled test sets
• Pipeline analysis — redundancy detection, orphaned chunks, grounding checks

Quick Start

from tracixx import RAGDebugger

debugger = RAGDebugger(
    catalog="main",
    schema="tracixx_debug",
    experiment_name="my-rag-v1"  # optional MLflow integration
)

# Wrap your RAG pipeline
with debugger.trace(query="What is Delta Lake?") as trace:
    # Log chunks from your document processor
    trace.log_chunks(chunks, strategy="recursive", chunk_size=512)

    # Log retrieved chunks with scores
    retrieved = vectorstore.similarity_search_with_score(query, k=5)
    trace.log_retrieval(chunks=retrieved, scores=scores, model="bge-large")

    # Log LLM response
    answer = llm.invoke(prompt)
    trace.log_response(answer=answer, prompt=prompt, tokens=usage)

LangChain Integration

Drop-in callback for LangChain chains:

from tracixx.integrations.langchain import RAGDebuggerCallback

chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(),
    callbacks=[RAGDebuggerCallback(debugger)]
)

Installation

pip install tracixx

Optional dependencies:

pip install tracixx[langchain]  # LangChain integration
pip install tracixx[mlflow]     # MLflow experiment tracking
pip install tracixx[spark]      # PySpark storage backend
pip install tracixx[all]        # All optional dependencies

Architecture

Storage: Delta tables in Unity Catalog (Databricks native)

Tables:

• traces — one row per query
• chunks — chunk metadata and text
• retrievals — retrieved chunks with scores
• llm_responses — generated answers and token usage

Analysis: Python-native diagnostics with no extra dependencies

Roadmap

• v0.1.0 (current): Core instrumentation + MLflow integration
• v0.2.0: Query replay, evaluation metrics, analysis modules
• Phase 2: Production monitoring, Genie Space integration, LLM-as-judge scoring

Development

For developers contributing to Tracixx:

git clone https://github.com/codezard/tracixx.git
cd tracixx
pip install -e ".[dev]"
pytest

License

MIT

Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines.

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions