A one-step Ragas cli tool to evaluate RAG apps
Project description
Ragas CLI
A one-step Ragas cli tool to evaluate QCAG testsets generated by RAG apps. (Q = Question, C = Contexts, A = Answer, G = Ground_truth)
Install with pip
pip install ragas langchain==0.0.354
then:
pip install ragas_once
Arguments
--model
: Specifies the model to use for evaluation.- Default value is "gpt-3.5-turbo". Langchain compatible.
--api_base
: Specifies the base URL for the API.- Default value is "https://api.openai.com/v1".
--api_key
: Specifies the API key to authenticate requests.- Not required if using psuedo-openai API server, e.g. vLLM, Fastchat, etc.
--embeddings
: Specifies the Huggingface embeddings model to use for evaluation.- Embeddings will run locally.
- Will use OpenAI embeddings if not set.
- Better set if using psuedo-openai API server.
--metrics
: Specifies the metrics to use for evaluation.- Will use Ragas default metrics if not set.
- Default metrics:
["answer_relevancy", "context_precision", "faithfulness", "context_recall", "context_relevancy"]
- Other metrics:
"answer_similarity", "answer_correctness"
--dataset
: Specifies the path to the dataset for evaluation.- Dataset format must meet RAGAS requirements.
- Will use fiqa dataset as demo if not set.
Usage
Fiqa dataset demo:
python3 -m ragas_once.cli --api_key "YOUR_OPENAI_API_KEY"
Evaluate with GPT-4 and BAAI/bge-small-en
embeddings
The huggingface embeddings will run locally, so Make sure your machine works and have sentence-transformers installed:
pip install sentence-transformers
Then run:
python3 -m ragas_once.cli --model "gpt-4" --api_key "YOUR_OPENAI_API_KEY" --embeddings "BAAI/bge-small-en" --dataset "path/to/dataset.csv"
Prepare Dataset
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
ragas_once-0.0.1.tar.gz
(8.3 kB
view hashes)
Built Distribution
Close
Hashes for ragas_once-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c604bf550c47864f30f658997ff8cf7034f582efff1d1dc9124c661bac947023 |
|
MD5 | 8f023e9e20bcec098b20dc19069f104c |
|
BLAKE2b-256 | dc58d28b11c5359386371a3a252b731f08e078ca18e888bcda2082dc2d3c5387 |