VeriScore 1.0.4

Pip package for Verifact

VeriScore

This is introduction for pip package of VeriScore. It have two type of method to extract claims by

1. prompting (Implemented)
2. fine-tuned model (in progress)

VeriScore consists of three parts 1) claim extraction 2) evidence searching and 3) claim verification. We provide an end-to-end pipeline to obtain the VeriScore, along with each of its components individually.

Install

1. Make a new Python 3.6+ environment using virtualenv or conda.
2. Install veriscore pacakge using pip
3. Download en_core_web_sm using spacy library

pip install --upgrade veriscore
python -m spacy download en_core_web_sm

Setup environment before running code

1. Download prompt folder that have txt file of prompt template (you can see prompt folder VeriScore's repository)
2. Set OpenAI or Claude API key to environment variable of bash for prompting approach

export OPENAI_API_KEY_PERSONAL={your_openai_api_key}
epport CLAUDE_API_KEY={your_claude_api_key}

3. Set SERPER API key to environment variable of bash for searching evidence

export SERPER_KEY_PRIVATE={your_serper_api_key}

Running VeriScore using a command line

This is an end-to-end pipeline for running VeriScore.

 python3 -m veriscore.veriscore --data_dir {data_dir} --input_file {input_file} --model_name_extraction {model_name_extraction} --model_name_verification {model_name_verification}

• data_dir: Directory containing input data. ./data by default.
• input_file: Name of input data file. It should be jsonl format where each line contains
  • question: query to ask
  • response: generated response from of question
  • model: name of model generate response
  • prompt_source: name of dataset provide question like FreshQA
• model_name_extraction: Name of model used for claim extraction. gpt-4-0125-preview by default.
• model_name_verification: Name of model used for claim verification. gpt-4o by default.

Other optional flags:

• output_dir: Directory for saving ouptut data. ./data by default.
• cache_dir: Directory for saving cache data. ./data/cache by default.
• label_n: This is type of label for claim verification. It could be 2 (binary) ro 3 (trinary)
  • 2: query to ask binary supported and unsupported
  • 3: query to ask trinay labels—supported, contradicted, and inconclusive
• search_res_num: A Hyperparameter for number of search result. 5 by default.

Saving output: input_file_name is file name removed jsonl from —-input_file

• extracted claims will be saved to output_dir/claims_{input_file_name}.jsonl.
• searched evidence will be saved to output_dir/evidence_{input_file_name}.jsonl.
• verified claims will be saved to output_dir/model_output/verification_{input_file_name}.jsonl.

Running individual part using a command line

Claim extraction:

 python3 -m veriscore.extract_claims --data_dir {data_dir} --input_file {input_file} --model_name {model_name} 

• input_file: Name of input data file. It should be jsonl format where each line contains
  • question: query to ask
  • response: generated response from of question
  • model: name of model generate response
  • prompt_source: name of dataset provide question like FreshQA
• model_name: Name of model used for claim extraction. gpt-4-0125-preview by default. output:

 {
  "question": question.strip(),
  "prompt_source": prompt_source,
  "response": response.strip(),
  "prompt_tok_cnt": prompt_tok_cnt,
  "response_tok_cnt": response_tok_cnt,
  "model": model,
  "abstained": False,  
  "claim_list": claim_list,
  "all_claim_lst": all_claim_lst
 }

Evidence searching:

 python3 -m veriscore.retrieval_evidence --data_dir {data_dir} --input_file {input_file}

• input_file: Name of input data file. It should be jsonl format where each line contains the keys of the output dictionary from the Claim extraction. output:

 {
  ...
  "claim_snippets_dict": dictionary for claim and list of searched evidence. each evidence have dictionary of {"title": title, "snippet": snippet, "link": link}
 }

Claim verification:

 python3 -m veriscore.verify_claims --data_dir {data_dir} --input_file {input_file} --model_name {model_name}

• input_file: Name of input data file. It should be jsonl format where each line contains the keys of the output dictionary from the Evidence searching. output:

 {
  ...
  "response": result of claim verification
 }