Skip to main content

🎯 Evret

A focused, lightweight retriever evaluation framework with standard Information Retrieval metrics

Python 3.10+ License: MIT Code style: black PyPI version Tests PRs Welcome

Evret brings standard Information Retrieval metrics to your recommendation, RAG and search systems. Evaluate retrievers with Hit Rate, Recall, Precision, MRR, NDCG, ERR, RBP and Average Precision in just a few lines of code. Built for simplicity, extensibility, and seamless integration with vector databases and AI frameworks.


🌟 Overview

Evret is a modern Python framework designed for evaluating retrieval systems in Information Retrieval pipelines and search applications. It provides:

  • Standard IR Metrics: Hit Rate, Recall, Precision, MRR, NDCG, ERR, RBP, and Average Precision
  • Judge-Based Matching: Token overlap, semantic, and LLM judges for text relevance
  • Vector Database Support: Native adapters for Qdrant and other vector databases
  • Framework Integration: Adapters for LangChain and LlamaIndex

🚀 Quick Start

Installation

pip install evret

For optional integrations:

pip install evret[all]

# Install specific integrations
pip install "evret[qdrant]"
pip install "evret[langchain]"
pip install "evret[semantic]"

5-Minute Evaluation

from evret import EvaluationDataset, Evaluator, HitRate, MRR, NDCG, TokenOverlapJudge

dataset = EvaluationDataset.from_json("eval_data.json")

evaluator = Evaluator(
    retriever=my_retriever,
    metrics=[HitRate(k=4), MRR(k=4), NDCG(k=4)],
    judge=TokenOverlapJudge(min_tokens=15, overlap_ratio=0.6),
)

results = evaluator.evaluate(dataset)
print(results.summary())

# Optional: Export results
results.to_json("results.json")
results.to_csv("results.csv")

🛠 Local Development Setup

Use uv for local development:

uv venv
source .venv/bin/activate
uv pip install -e .

Install all optional integrations:

uv pip install -e ".[all]"

Run the default test suite:

pytest

See tests/README.md for coverage areas, optional integration tests, and test setup notes.


📊 Metrics

Evret supports all standard Information Retrieval metrics:

Metric Description Use Case
Hit Rate@k % of queries with at least one relevant doc in top-k Binary relevance, recall-focused
Recall@k % of relevant docs found in top-k Comprehensive retrieval
Precision@k % of top-k results that are relevant Precision-focused systems
MRR@k Mean Reciprocal Rank of first relevant doc Single-answer retrieval
NDCG@k Normalized Discounted Cumulative Gain Rank-aware binary relevance quality
ERR@k Expected Reciprocal Rank with cascade satisfaction Graded relevance and user satisfaction
RBP@k Rank-Biased Precision with tunable persistence User patience and position-weighted quality
Average Precision@k Area under precision-recall curve Overall ranking quality

📋 Evaluation Datasets

Create evaluation datasets with queries and expected answers for judge-based evaluation:

from evret import EvaluationDataset, QueryExample, DocumentExample

dataset = EvaluationDataset(
    documents=[
        DocumentExample(doc_id="doc_1", text="Python uses pip for packages."),
        DocumentExample(doc_id="doc_2", text="Virtual environments isolate dependencies."),
    ],
    queries=[
        QueryExample(
            query_id="q1",
            query_text="How to install Python packages?",
            expected_answers=["pip install"]  # Judge matches this against retrieved text
        )
    ]
)

Load datasets from JSON or CSV files:

dataset = EvaluationDataset.from_json("eval_data.json")
dataset = EvaluationDataset.from_csv("eval_data.csv")

For detailed dataset format documentation and more examples, see the Dataset Format Guide


⚖️ Judges

Judges decide whether retrieved text matches the expected text in your evaluation dataset. Evaluator uses TokenOverlapJudge() by default, and you can pass judge= when you want explicit matching behavior.

from evret import Evaluator, HitRate, Recall
from evret.judges import TokenOverlapJudge

evaluator = Evaluator(
    retriever=my_retriever,
    metrics=[HitRate(k=4), Recall(k=4)],
    judge=TokenOverlapJudge(min_tokens=15, overlap_ratio=0.6),
)

Use SemanticJudge for embedding similarity and LLMJudge for LLM-provider based judgment.

from evret.judges import LLMJudge, SemanticJudge

semantic_judge = SemanticJudge(threshold=0.75)
llm_judge = LLMJudge(provider="openai", model="gpt-4o-mini")

🔌 Integrations

Qdrant Vector Database

from evret.retrievers import QdrantRetriever

retriever = QdrantRetriever(
    collection_name="docs",
    query_encoder=embed_query,
    url="http://localhost:6333",
    id_field="doc_id",
)

LangChain Integration

from evret.integrations import LangChainRetrieverAdapter

# Wrap any Evret retriever for use in LangChain
lc_retriever = LangChainRetrieverAdapter(evret_retriever=retriever, k=5)
docs = lc_retriever.invoke("what is information retrieval?")

⚙️ Configuration

Logging

Evret uses standard Python logging. Set the log level via environment variable:

export EVRET_LOG_LEVEL=INFO    # INFO, DEBUG, WARNING, ERROR

Or configure programmatically:

import logging
logging.basicConfig(level=logging.INFO, format="%(levelname)s [%(name)s] %(message)s")

Default level is WARNING to avoid spamming logs in production.


🧪 Testing

Run the default suite:

pytest

Run Docker-backed integration tests:

EVRET_RUN_INTEGRATION=1 pytest -m integration

More details are in tests/README.md.


📚 Documentation

Evret docs use MkDocs with Material theme.

Install docs dependencies:

uv pip install -e ".[docs]"

Run docs locally:

mkdocs serve

Build static docs:

mkdocs build

📄 License

MIT License - see LICENSE for details.


🤝 Contributing

Contributions are welcome! Please read our Contributing Guide for details on our code of conduct and the process for submitting pull requests.


📚 Citation

If you use Evret in your research, please cite:

@software{evret2026,
  title={Evret: A Focused Retriever Evaluation Framework},
  author={lucifertrj},
  year={2026},
}

**Built with ❤️ for the Information Retrieval community**

Metadata

Release files for evret 0.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for evret 0.0.4
File Size Uploaded Provenance Actions
evret-0.0.4.tar.gz 46.9 kB Without provenance Details for evret-0.0.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for evret 0.0.4
File Interpreter ABI Platform Actions
evret-0.0.4-py3-none-any.whl Python 3 none any Details for evret-0.0.4-py3-none-any.whl

Total release size: 111.9 kB

Release files / evret-0.0.4.tar.gz

Download URL evret-0.0.4.tar.gz
Size 46.9 kB
Tags Source
SHA-256 checksum
How to use checksums
2a4e03b053b02a03601692280d8cfd463fae6d90338bf2735a5b713ab1b854f6
BLAKE2b-256 checksum
How to use checksums
c937303757df69b3565eed014d6312c40cb5720cbfe702f1407daefc9ab4ce86
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.1.4 CPython/3.11.13 Darwin/25.0.0

Release files / evret-0.0.4-py3-none-any.whl

Download URL evret-0.0.4-py3-none-any.whl
Size 65.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c571cc04e3216ee0593edd8de90f0b5881b8458e068fef6e0f1dade7ff6a03af
BLAKE2b-256 checksum
How to use checksums
f30b5bc7adc156d7d15745c4f18120e7cfbf7c835dc7b98b2d5fabbed183f779
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.1.4 CPython/3.11.13 Darwin/25.0.0

Security for evret 0.0.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page