Skip to main content

A one-step Ragas cli tool to evaluate RAG apps

Project description

Ragas CLI

A one-step Ragas cli tool to evaluate QCAG testsets generated by RAG apps. (Q = Question, C = Contexts, A = Answer, G = Ground_truth)

Install with pip

pip install ragas langchain==0.0.354

then:

pip install ragas_once

Arguments

  • --model: Specifies the model to use for evaluation.
    • Default value is "gpt-3.5-turbo". Langchain compatible.
  • --api_base: Specifies the base URL for the API.
  • --api_key: Specifies the API key to authenticate requests.
    • Not required if using psuedo-openai API server, e.g. vLLM, Fastchat, etc.
  • --embeddings: Specifies the Huggingface embeddings model to use for evaluation.
    • Embeddings will run locally.
    • Will use OpenAI embeddings if not set.
    • Better set if using psuedo-openai API server.
  • --metrics: Specifies the metrics to use for evaluation.
    • Will use Ragas default metrics if not set.
    • Default metrics: ["answer_relevancy", "context_precision", "faithfulness", "context_recall", "context_relevancy"]
    • Other metrics: "answer_similarity", "answer_correctness"
  • --dataset: Specifies the path to the dataset for evaluation.
    • Dataset format must meet RAGAS requirements.
    • Will use fiqa dataset as demo if not set.

Usage

Fiqa dataset demo:

python3 -m ragas_once.cli --api_key "YOUR_OPENAI_API_KEY"

Evaluate with GPT-4 and BAAI/bge-small-en embeddings

The huggingface embeddings will run locally, so Make sure your machine works and have sentence-transformers installed:

pip install sentence-transformers

Then run:

python3 -m ragas_once.cli --model "gpt-4" --api_key "YOUR_OPENAI_API_KEY" --embeddings "BAAI/bge-small-en" --dataset "path/to/dataset.csv"

Prepare Dataset

See Ragas documentation

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ragas_once-0.0.1.tar.gz (8.3 kB view hashes)

Uploaded Source

Built Distribution

ragas_once-0.0.1-py3-none-any.whl (8.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page