ragret 0.1.2

Lightweight evaluation framework for Retrieval Augmented Generation systems, focused on simplicity and long-term consistency.

RAG evaluation with fewer regrets.

     

ragret is a lightweight, stable evaluation framework for Retrieval-Augmented Generation (RAG) systems that is designed for long-term consistency and only the necessary structural updates in mind.
Its goal is simplicity: small, modular metrics that are easy to understand, extend, and integrate into existing pipelines. It was created out of the frustration with other frameworks constantly changing, making code from one version to the next obsolete and difficult to migrate. With ragret, the focus is clear: simple, implement-as-you-go metrics that you can rely on without having to rewrite your established code or digging through docs to figure out what changed overnight in your favorite framework.

Metrics

ragret provides evaluation metrics for assessing different aspects of RAG system performance.
It includes both LLM-based and non-LLM-based metrics, which are described with more detail in METRICS

• AnswerRelevancy
• Faithfulness
• ContextPrecision
• ContextRecall
• CosineSimilarity
• F1Score
• ProductRelevancy Custom (for systems related to product recommendation)

Installation

Use pip to install the package

pip install ragret

Or

pip install git+https://github.com/christopherkormpos/ragret.git

Or you can clone the repository locally:

git clone https://github.com/christopherkormpos/ragret.git
cd ragret

Supported providers

ragret currently supports only two LLM providers for generation and embeddings.
The default models for text generation are gpt-4.1 for OpenAI and gemma3:4b for Ollama.
For vector embeddings, the default models are text-embeddings-small-3 for OpenAI and nomic-embed-text for Ollama.

Basic Configuration

You will need to create a .env file and set your enviromental variable "API_KEY" to your providers API key (if you are using external LLMs)

API_KEY=your-api-key-here

Or you can pass it directly during class initialization.

Faithfulness(provider="openai", api_key="your-api-key-here")

Usage

All metrics are exposed on upper level. Therefore they can be imported as such:

from ragret import (
  AnswerRelevancy
  Faithfulness
  ContextPrecision
  ContextRecall
  CosineSimilarity
  F1Score
  ProductRelevancy
)

Usage may vary depending on what you want to do. There are two ways to evaluate your dataset.

Use case 1 (Common - Simpler - Faster): Use the Evaluator class

# Import the metrics we want to use to evaluate our dataset, evaluator, and example dataset
from ragret import ContextRecall, ContextPrecision, AnswerRelevancy
from ragret.evaluators import Evaluator
from ragret.datasets import example_dataset
import pandas as pd

# Initialize metric classes with the desired provider.
# Other optional parameters:
# - api_key: provide your API key directly
# - ollama_url: for local LLM models
# - model: the LLM model name
# - embedding_model: the embedding model to use
context_recall = ContextRecall(provider="openai")
context_precision = ContextPrecision(provider="openai")
answer_relevancy = AnswerRelevancy(provider="openai")

# Create the evaluator with the dataset
# Use the calculate() method to evaluate the dataset with the selected metrics
results = Evaluator(example_dataset).calculate(
  context_recall,
  answer_relevancy,
  context_precision
  )

# Finally convert the results into a DataFrame and save the output in the current working directory.
df = pd.DataFrame(results)
df.to_csv("evaluation_results.csv", index=False)
print("Results saved to evaluation_results.csv")

With the help of pandas, we convert our results into a DataFrame and save the output in the current working directory.

Note: It’s important that the dataset is structured like the example below so that the Evaluator class can work correctly and produce results.

[
    {
        "user_query": "User Question 1",
        "retrieved_documents": ["Retrieved document text 1", 
                                "Retrieved document text 2", 
                                "Retrieved document text 3"],
        "llm_answer": "LLM Answer for Question 1",
        "ground_truth": "Ground-truth (reference) answer for Question 1"
    },
    {
        "user_query": "User Question 2",
        "retrieved_documents": ["Retrieved document text 1", 
                                "Retrieved document text 2", 
                                "Retrieved document text 3"],
        "llm_answer": "LLM Answer for Question 2",
        "ground_truth": "Ground-truth (reference) answer for Question 2"
    },...
]

Use case 2 : Use the metrics classes directly

# Import the metrics we want to use to evaluate our dataset, evaluator, and example dataset
from ragret import ContextRecall, ContextPrecision, AnswerRelevancy
from ragret.datasets import example_dataset
import pandas as pd

# Initialize metric classes with the desired provider.
# Other optional parameters:
# - api_key: provide your API key directly
# - ollama_url: for local LLM models
# - model: the LLM model name
# - embedding_model: the embedding model to use
context_recall = ContextRecall(provider="openai")
context_precision = ContextPrecision(provider="openai")
answer_relevancy = AnswerRelevancy(provider="openai")

results = []
# Use a simple for loop to evaluate the dataset with the selected metrics
for i,record in enumerate(example_dataset):
    print(f"Evaluating record {i+1} of {len(example_dataset)}")

    recall_result = context_recall.score(
        retrieved_documents=record["retrieved_documents"],
        ground_truth=record["ground_truth"]
    )["score"]
    
    precision_result = context_precision.score(
        user_query=record["user_query"],
        retrieved_documents=record["retrieved_documents"]
    )["score"]
    
    answer_relevancy_result = answer_relevancy.score(
        user_query=record["user_query"],
        llm_answer=record["llm_answer"])["score"]

    results.append({
        "context_precision": precision_result,
        "context_recall": recall_result,
        "answer_relevancy":answer_relevancy_result
    })

# Finally convert the results into a DataFrame and save the output in the current working directory.
df = pd.DataFrame(results)
df.to_csv("evaluation_results.csv", index=False)
print("Results saved to evaluation_results.csv")

Contact

If you encounter any issues or bugs with the application, feel free to reach out to me:

• Email: [e-mail]
• GitHub: https://github.com/christopherkormpos
• LinkedIn: LinkedIn Profile

License

This project is licensed under the MIT License - see the LICENSE file for details.