A one-step Ragas cli tool to evaluate RAG apps
Project description
Ragas CLI
A one-step Ragas cli tool to evaluate QCAG testsets generated by RAG apps. (Q = Question, C = Contexts, A = Answer, G = Ground_truth)
Install with pip
pip install ragas langchain==0.0.354
then:
pip install ragas_once
Arguments
--model
: Specifies the model to use for evaluation.- Default value is "gpt-3.5-turbo". Langchain compatible.
--api_base
: Specifies the base URL for the API.- Default value is "https://api.openai.com/v1".
--api_key
: Specifies the API key to authenticate requests.- Not required if using psuedo-openai API server, e.g. vLLM, Fastchat, etc.
--embeddings
: Specifies the Huggingface embeddings model to use for evaluation.- Embeddings will run locally.
- Will use OpenAI embeddings if not set.
- Better set if using psuedo-openai API server.
--metrics
: Specifies the metrics to use for evaluation.- Will use Ragas default metrics if not set.
- Default metrics:
["answer_relevancy", "context_precision", "faithfulness", "context_recall", "context_relevancy"]
- Other metrics:
"answer_similarity", "answer_correctness"
--dataset
: Specifies the path to the dataset for evaluation.- Dataset format must meet RAGAS requirements.
- Will use fiqa dataset as demo if not set.
Usage
Fiqa dataset demo:
python3 -m ragas_once.cli --api_key "YOUR_OPENAI_API_KEY"
Evaluate with GPT-4 and BAAI/bge-small-en
embeddings
The huggingface embeddings will run locally, so Make sure your machine works and have sentence-transformers installed:
pip install sentence-transformers
Then run:
python3 -m ragas_once.cli --model "gpt-4" --api_key "YOUR_OPENAI_API_KEY" --embeddings "BAAI/bge-small-en" --dataset "path/to/dataset.csv"
Prepare Dataset
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
ragas_once-0.0.1.tar.gz
(8.3 kB
view details)
Built Distribution
File details
Details for the file ragas_once-0.0.1.tar.gz
.
File metadata
- Download URL: ragas_once-0.0.1.tar.gz
- Upload date:
- Size: 8.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8fd7ccbb9f4899eb53ec6ff3440a5e6de6e533b77b7674a95de8639af5dc38c6 |
|
MD5 | 08b73b92232421044a5bc4b68fda13b8 |
|
BLAKE2b-256 | 45603fb9735070f16cbe158b7e1352161b3d3f9fdbf673dc4851734975486c44 |
File details
Details for the file ragas_once-0.0.1-py3-none-any.whl
.
File metadata
- Download URL: ragas_once-0.0.1-py3-none-any.whl
- Upload date:
- Size: 8.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c604bf550c47864f30f658997ff8cf7034f582efff1d1dc9124c661bac947023 |
|
MD5 | 8f023e9e20bcec098b20dc19069f104c |
|
BLAKE2b-256 | dc58d28b11c5359386371a3a252b731f08e078ca18e888bcda2082dc2d3c5387 |