halludetector 0.0.6

Hallucination detection package

This project implements generic approaches for hallucination detection.

The Detector base class implements the building blocks to detect hallucinations and score them.

ask_llm - method to request a response from an LLM via a prompt

extract_triplets - method to extract subject, predicate, object from a text.

extract_sentences - method to split a text into sentences using spacy

generate_question - method to generate a question from a text

retrieve - method to retrieve information from google via the serper api

check - method to check if the claims contain hallucinations

similarity_bertscore - method to check the similarity between texts via bertscore

similarity_ngram - method to check the similarity between texts via ngram model

You can implement any custom detector and combine all the available methods from above.

Installation

Use a conda environment and install the followings.

pip install -e .
pip install -r requirements.txt

python3 -m spacy download en_core_web_sm

Export envs for openai and google wrapper

export OPENAI_API_KEY=
export SERPER_API_KEY=

Usage

as server

python3 server.py

Go to http://127.0.0.1:5000 and use the app.

as library

from openai import OpenAI
from halludetector import calculate_score

client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

# prompt - the prompt as text
# the question you want to ask as text
# 5 times to ask the llm if the completion contains hallucinations

score, completion, explanations = calculate_score(client, prompt, question, 5)

print(score)

from cli

python3 scorer.py --file data/questions.json --config config.json

Configuration for ChainPoll

The file /data/prompt.txt contains a prompt template that is used.

Feel free to update it.

Allowed variables:

{completion} for the answer to the question.

{question} the initial question.

The file config.json contains the needed configuration for the library.