Skip to main content

A one-step Ragas cli tool to evaluate RAG apps

Project description

Ragas CLI

A one-step Ragas cli tool to evaluate QCAG testsets generated by RAG apps. (Q = Question, C = Contexts, A = Answer, G = Ground_truth)

Install with pip

pip install ragas langchain==0.0.354

then:

pip install ragas_once

Arguments

  • --model: Specifies the model to use for evaluation.
    • Default value is "gpt-3.5-turbo". Langchain compatible.
  • --api_base: Specifies the base URL for the API.
  • --api_key: Specifies the API key to authenticate requests.
    • Not required if using psuedo-openai API server, e.g. vLLM, Fastchat, etc.
  • --embeddings: Specifies the Huggingface embeddings model to use for evaluation.
    • Embeddings will run locally.
    • Will use OpenAI embeddings if not set.
    • Better set if using psuedo-openai API server.
  • --metrics: Specifies the metrics to use for evaluation.
    • Will use Ragas default metrics if not set.
    • Default metrics: ["answer_relevancy", "context_precision", "faithfulness", "context_recall", "context_relevancy"]
    • Other metrics: "answer_similarity", "answer_correctness"
  • --dataset: Specifies the path to the dataset for evaluation.
    • Dataset format must meet RAGAS requirements.
    • Will use fiqa dataset as demo if not set.

Usage

Fiqa dataset demo:

python3 -m ragas_once.cli --api_key "YOUR_OPENAI_API_KEY"

Evaluate with GPT-4 and BAAI/bge-small-en embeddings

The huggingface embeddings will run locally, so Make sure your machine works and have sentence-transformers installed:

pip install sentence-transformers

Then run:

python3 -m ragas_once.cli --model "gpt-4" --api_key "YOUR_OPENAI_API_KEY" --embeddings "BAAI/bge-small-en" --dataset "path/to/dataset.csv"

Prepare Dataset

See Ragas documentation

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ragas_once-0.0.1.tar.gz (8.3 kB view details)

Uploaded Source

Built Distribution

ragas_once-0.0.1-py3-none-any.whl (8.8 kB view details)

Uploaded Python 3

File details

Details for the file ragas_once-0.0.1.tar.gz.

File metadata

  • Download URL: ragas_once-0.0.1.tar.gz
  • Upload date:
  • Size: 8.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.5

File hashes

Hashes for ragas_once-0.0.1.tar.gz
Algorithm Hash digest
SHA256 8fd7ccbb9f4899eb53ec6ff3440a5e6de6e533b77b7674a95de8639af5dc38c6
MD5 08b73b92232421044a5bc4b68fda13b8
BLAKE2b-256 45603fb9735070f16cbe158b7e1352161b3d3f9fdbf673dc4851734975486c44

See more details on using hashes here.

File details

Details for the file ragas_once-0.0.1-py3-none-any.whl.

File metadata

  • Download URL: ragas_once-0.0.1-py3-none-any.whl
  • Upload date:
  • Size: 8.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.5

File hashes

Hashes for ragas_once-0.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 c604bf550c47864f30f658997ff8cf7034f582efff1d1dc9124c661bac947023
MD5 8f023e9e20bcec098b20dc19069f104c
BLAKE2b-256 dc58d28b11c5359386371a3a252b731f08e078ca18e888bcda2082dc2d3c5387

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page