Ragas CLI

A one-step Ragas cli tool to evaluate QCAG testsets generated by RAG apps. (Q = Question, C = Contexts, A = Answer, G = Ground_truth)

Install with pip

pip install ragas langchain==0.0.354

then:

pip install ragas_once

Arguments

• --model: Specifies the model to use for evaluation.
  • Default value is "gpt-3.5-turbo". Langchain compatible.
• --api_base: Specifies the base URL for the API.
  • Default value is "https://api.openai.com/v1".
• --api_key: Specifies the API key to authenticate requests.
  • Not required if using psuedo-openai API server, e.g. vLLM, Fastchat, etc.
• --embeddings: Specifies the Huggingface embeddings model to use for evaluation.
  • Embeddings will run locally.
  • Will use OpenAI embeddings if not set.
  • Better set if using psuedo-openai API server.
• --metrics: Specifies the metrics to use for evaluation.
  • Will use Ragas default metrics if not set.
  • Default metrics: ["answer_relevancy", "context_precision", "faithfulness", "context_recall", "context_relevancy"]
  • Other metrics: "answer_similarity", "answer_correctness"
• --dataset: Specifies the path to the dataset for evaluation.
  • Dataset format must meet RAGAS requirements.
  • Will use fiqa dataset as demo if not set.

Usage

Fiqa dataset demo:

python3 -m ragas_once.cli --api_key "YOUR_OPENAI_API_KEY"

Evaluate with GPT-4 and BAAI/bge-small-en embeddings

The huggingface embeddings will run locally, so Make sure your machine works and have sentence-transformers installed:

pip install sentence-transformers

Then run:

python3 -m ragas_once.cli --model "gpt-4" --api_key "YOUR_OPENAI_API_KEY" --embeddings "BAAI/bge-small-en" --dataset "path/to/dataset.csv"

Prepare Dataset

See Ragas documentation