Skip to main content

No project description provided

Project description

Overview

Ariadne AI is an open-source library for evaluating text summarization and retireved-augmented-generation (RAG) chatbots without the necessity for human-annotated reference summaries. Each evaluator is paired with an explanation that help developers evaluate their LLMs and detect the reason for their failure cases. Our approach leverages LLM's reasoning to provide explanations for the failures.

Installation

pip install ariadne-ai

or

 poetry run python run_experiment_rag.py
 poetry run python run_experiment_summarization.py

or

see the provided notebook examples

Text Summarization

Text Summarization QAG Approach

For text summarization, a question-answer generation (QAG) framework has been developed, which allows us to pinpoint failure cases in production without human-annotated reference summaries. Here is a breakdown of our approach:

  1. Question Generation: The LLM formulates closed-ended (Yes/No) questions drawing from both the summary and the main document.
  2. Summary-based Answers: An LLM answerer generator responds to these questions using only the summary as a reference. The potential responses include "Yes," "No," and "Unknown."
  3. Document-based Answers: Similarly, the LLM answerer generator answers the same set of questions, but this time, it references the primary document. Possible responses remain "Yes," "No," and "Unknown."
  4. Evaluation Metrics: The evaluation metrics assessing the consistency between the summary-based and document-based summaries are computed to draw conclusions.

The following failure are detected based on the above approach:

  • Hallucination Failure: A hallucination failure occurs when a question gets a 'Yes/No' answer based on the summary but receives an 'Unknown' answer based on the original document.

  • Contradiction Failure: A contradiction failure is detected when at least one question is answered 'Yes' based on the summary, but 'No' when based on the full document, or vice-versa.

  • Non-informativeness Failure: A non-informativeness failure occurs when at least one question is answered as 'Unknown' based on the summary but a definitive 'Yes/No' based on the original document.

Retrived Augmented Generation (RAG)

Here is a breakdown of our approach:

  1. Evaluation: The LLM determines whether a critirion is met, leveraging its reasoning
  2. Explanation: The LLM explains the reasoning behind its decision, providing clarity regarding the failure cases.

The following failure cases are detected:

  1. Faithfulness Failure: A faithfulness failure occurs if the response cannot be inferred purely from the context provided.
  2. Context Relevance Failure: A context relevance failure (bad retrieval) occures if the user's query cannot be answered purely from the retrieved context.
  3. Answer Relevance Failure: An answer relevacne failure occurs if the response does not answer the question.

Contribution

Please feel free to reach out to shiv@athina.ai if you would like to contribute. You could find more on how you could integrate the evaluations in your product here: https://docs.athina.ai.

License

Apache License 2.0

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ariadne_ai-0.1.12.tar.gz (20.6 kB view details)

Uploaded Source

Built Distribution

ariadne_ai-0.1.12-py3-none-any.whl (39.6 kB view details)

Uploaded Python 3

File details

Details for the file ariadne_ai-0.1.12.tar.gz.

File metadata

  • Download URL: ariadne_ai-0.1.12.tar.gz
  • Upload date:
  • Size: 20.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.6.1 CPython/3.9.16 Darwin/23.0.0

File hashes

Hashes for ariadne_ai-0.1.12.tar.gz
Algorithm Hash digest
SHA256 067b94a758d691ba819f54a87ae218ed63a9b509a79d82770c85f62595f80335
MD5 b94376a5867fbf16e18367025044c714
BLAKE2b-256 45736eeec8a9b2ba7cfb9758ee18ba1959cc4db31a243ba62309ad9c955b34b6

See more details on using hashes here.

File details

Details for the file ariadne_ai-0.1.12-py3-none-any.whl.

File metadata

  • Download URL: ariadne_ai-0.1.12-py3-none-any.whl
  • Upload date:
  • Size: 39.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.6.1 CPython/3.9.16 Darwin/23.0.0

File hashes

Hashes for ariadne_ai-0.1.12-py3-none-any.whl
Algorithm Hash digest
SHA256 4fc17e7762f56fcbb1506fc206cc459a7703046e610c2a85e8a57150721fefc5
MD5 92566b31fda9de51ac397ee860d8c593
BLAKE2b-256 1a483a0217c7e32b9cff23590842fa25860066df51d2f10f4d0d942ea1db2575

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page