Skip to main content

RAG Evaluator

Overview

RAG Evaluator is a Python library for evaluating Retrieval-Augmented Generation (RAG) systems. It provides various metrics to evaluate the quality of generated text against reference text.

Installation

You can install the library using pip:

pip install rag-evaluate

Usage: Here's how to use the RAG Evaluator library:

from rag_evaluate import RAG_Evaluator

Initialize the evaluator

evaluator = RAG_Evaluator()

Input data

question = "What are the causes of climate change?." generated_text = "Climate change is caused by human activities." context = "Human activities such as burning fossil fuels cause climate change."

Evaluate the response

bleu_score = evaluator.bleu_score(question, generated_text, context)
rouge_score = evaluator.rouge_score(question, generated_text, context)
bert_score = evaluator.bert_score(question, generated_text, context)

Print the results

print(bleu_score)
print(rouge_score)  
print(bert_score)  

The RAG Evaluator provides the following metrics:

BLEU (0-100): Measures the overlap between the generated output and reference text based on n-grams.

0-20: Low similarity, 20-40: Medium-low, 40-60: Medium, 60-80: High, 80-100: Very high.

ROUGE-1 (0-1): Measures the overlap of unigrams between the generated output and reference text.

0.0-0.2: Poor overlap, 0.2-0.4: Fair, 0.4-0.6: Good, 0.6-0.8: Very good, 0.8-1.0: Excellent.

BERT Score (0-1): Evaluates the semantic similarity using BERT embeddings (Precision, Recall, F1).

0.0-0.5: Low similarity, 0.5-0.7: Moderate, 0.7-0.8: Good, 0.8-0.9: High, 0.9-1.0: Very high.

Contributer

Biplab Sil.

Metadata

Release files for rag-evaluate 0.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for rag-evaluate 0.5.0
File Size Uploaded Provenance Actions
rag_evaluate-0.5.0.tar.gz 2.5 kB Without provenance Details for rag_evaluate-0.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for rag-evaluate 0.5.0
File Interpreter ABI Platform Actions
rag_evaluate-0.5.0-py3-none-any.whl Python 3 none any Details for rag_evaluate-0.5.0-py3-none-any.whl

Total release size: 5.2 kB

Release files / rag_evaluate-0.5.0.tar.gz

Download URL rag_evaluate-0.5.0.tar.gz
Size 2.5 kB
Tags Source
SHA-256 checksum
How to use checksums
360c3683ef4b303aca26885538ff2b41759b574d40bed1bab7dfddbcd6842949
BLAKE2b-256 checksum
How to use checksums
dd7be0e26464162454442d06ddd7b97cacc0586e898045c05e1f3047be5ae6a0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.2

Release files / rag_evaluate-0.5.0-py3-none-any.whl

Download URL rag_evaluate-0.5.0-py3-none-any.whl
Size 2.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f3ccca3867ed727fc98440525783978ac2ce701deb65a5f761cbce618ea507e2
BLAKE2b-256 checksum
How to use checksums
5946416e08b3cd2cc605e49d253fca33f161d4e75d0dddf09424995306cafae3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.13.2

Security for rag-evaluate 0.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page