ragas

Supercharge Your LLM Application Evaluations 🚀

Documentation | Quick start | Join Discord | Blog | NewsLetter | Careers

Objective metrics, intelligent test generation, and data-driven insights for LLM apps

Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows. Don't have a test dataset ready? We also do production-aligned test set generation.

[!NOTE] Need help setting up Evals for your AI application? We'd love to help! We are conducting Office Hours every week. You can sign up here.

Key Features

🎯 Objective Metrics: Evaluate your LLM applications with precision using both LLM-based and traditional metrics.
🧪 Test Data Generation: Automatically create comprehensive test datasets covering a wide range of scenarios.
🔗 Seamless Integrations: Works flawlessly with popular LLM frameworks like LangChain and major observability tools.
📊 Build feedback loops: Leverage production data to continually improve your LLM applications.

:shield: Installation

Pypi:

pip install ragas

Alternatively, from source:

pip install git+https://github.com/vibrantlabsai/ragas

:fire: Quickstart

Clone a Complete Example Project

The fastest way to get started is to use the ragas quickstart command:

# List available templates
ragas quickstart

# Create a RAG evaluation project
ragas quickstart rag_eval

# Specify where you want to create it.
ragas quickstart rag_eval -o ./my-project

Available templates:

rag_eval - Evaluate RAG systems

Coming Soon:

agent_evals - Evaluate AI agents
benchmark_llm - Benchmark and compare LLMs
prompt_evals - Evaluate prompt variations
workflow_eval - Evaluate complex workflows

Evaluate your LLM App

This is a simple example evaluating a summary for accuracy:

import asyncio
from ragas.metrics.collections import AspectCritic
from ragas.llms import llm_factory

# Setup your LLM
llm = llm_factory("gpt-4o")

# Create a metric
metric = AspectCritic(
    name="summary_accuracy",
    definition="Verify if the summary is accurate and captures key information.",
    llm=llm
)

# Evaluate
test_data = {
    "user_input": "summarise given text\nThe company reported an 8% rise in Q3 2024, driven by strong performance in the Asian market. Sales in this region have significantly contributed to the overall growth. Analysts attribute this success to strategic marketing and product localization. The positive trend in the Asian market is expected to continue into the next quarter.",
    "response": "The company experienced an 8% increase in Q3 2024, largely due to effective marketing strategies and product adaptation, with expectations of continued growth in the coming quarter.",
}

score = await metric.ascore(
    user_input=test_data["user_input"],
    response=test_data["response"]
)
print(f"Score: {score.value}")
print(f"Reason: {score.reason}")

Note: Make sure your OPENAI_API_KEY environment variable is set.

Find the complete Quickstart Guide

Want help in improving your AI application using evals?

In the past 2 years, we have seen and helped improve many AI applications using evals. If you want help with improving and scaling up your AI application using evals.

🔗 Book a slot or drop us a line: founders@vibrantlabs.com.

🫂 Community

If you want to get more involved with Ragas, check out our discord server. It's a fun community where we geek out about LLM, Retrieval, Production issues, and more.

Contributors

+----------------------------------------------------------------------------+
|     +----------------------------------------------------------------+     |
|     | Developers: Those who built with `ragas`.                      |     |
|     | (You have `import ragas` somewhere in your project)            |     |
|     |     +----------------------------------------------------+     |     |
|     |     | Contributors: Those who make `ragas` better.       |     |     |
|     |     | (You make PR to this repo)                         |     |     |
|     |     +----------------------------------------------------+     |     |
|     +----------------------------------------------------------------+     |
+----------------------------------------------------------------------------+

We welcome contributions from the community! Whether it's bug fixes, feature additions, or documentation improvements, your input is valuable.

Fork the repository
Create your feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

🔍 Open Analytics

At Ragas, we believe in transparency. We collect minimal, anonymized usage data to improve our product and guide our development efforts.

✅ No personal or company-identifying information

✅ Open-source data collection code

✅ Publicly available aggregated data

To opt-out, set the RAGAS_DO_NOT_TRACK environment variable to true.

Cite Us

@misc{ragas2024,
  author       = {VibrantLabs},
  title        = {Ragas: Supercharge Your LLM Application Evaluations},
  year         = {2024},
  howpublished = {\url{https://github.com/vibrantlabsai/ragas}},
}

Release files for ragas 0.4.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ragas 0.4.3
File	Size	Uploaded
ragas-0.4.3.tar.gz	44.0 MB	Jan 13, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for ragas 0.4.3
File	Interpreter	ABI	Platform	Reset
ragas-0.4.3-py3-none-any.whl 466.5 kB Jan 13, 2026	Python 3	none	any	Details

Total release size: 44.5 MB

Release files / ragas-0.4.3.tar.gz

Download URL	ragas-0.4.3.tar.gz
Size	44.0 MB
Tags	Source
SHA-256 checksum How to use checksums	`1eb1f61dbc8613ad014fdb8d630cbe9a1caec1ea01664a106993cb756128c001`
BLAKE2b-256 checksum How to use checksums	`d2bc3234517692ac0ffae1ec2ec940992e4057844c49ee6c51c07ce385bb98f1`
Upload date	Jan 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.7`

Release files / ragas-0.4.3-py3-none-any.whl

Download URL	ragas-0.4.3-py3-none-any.whl
Size	466.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`ef1d75f674c294e9a6e7d8e9ad261b6bf4697dad1c9cbd1a756ba7a6b4849a38`
BLAKE2b-256 checksum How to use checksums	`4de01fecd22c93d3ed66453cbbdefd05528331af4d33b2b76a370d751231912c`
Upload date	Jan 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.7`