verifastscore 0.1.5

Fast, end-to-end factuality evaluation for long-form LLM responses.

🔍⚡ VeriFastScore

VeriFastScore is a fast and efficient factuality evaluation tool that jointly extracts and verifies fine-grained factual claims from long-form LLM-generated responses, conditioned on evidence collated from google search results retrieved using SERPER API.

This repository packages VeriFastScore as a pip-installable Python package with a command-line interface (verifastscore), simplifying usage and deployment.

---

Coming soon: Dynamic Batching to further imrpove speedups and add support for multi-GPU usage

Important Links

• Paper
• Model
• Dataset

---

📦 Installation

VeriFastScore supports both CPU and GPU setups. Please follow the appropriate instructions below:

▶️ CPU Installation [Default]

To install VeriFastScore for CPU-only usage:

pip install verifastscore
python3 -m spacy download en_core_web_sm

▶️ GPU Installation (with FlashAttention) [Recommended]

If you want GPU acceleration with FlashAttention, you must install compatible versions of torch and flash-attn manually before installing verifastscore.

# Install CUDA-enabled PyTorch (adjust for your CUDA version)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

# Install flash-attn (manually, requires build tools and matching CUDA)
pip install flash-attn --no-build-isolation

# Then install verifastscore
pip install verifastscore

# Finally, download the spaCy model
python3 -m spacy download en_core_web_sm

▶️ Development install (local)

git clone https://github.com/rishanthrajendhran/verifastscore.git
cd verifastscore
pip install -e .
python3 -m spacy download en_core_web_sm

---

🔐 Set Your SERPER API Key

VeriFastScore retrieves external evidence using Serper.dev. You'll need to set your API key in the environment:

💻 Linux/macOS

export SERPER_KEY_PRIVATE="your-key-here"

Add to ~/.bashrc or ~/.zshrc for permanence:

echo 'export SERPER_KEY_PRIVATE="your-key-here"' >> ~/.bashrc
source ~/.bashrc

🪟 Windows CMD

set SERPER_KEY_PRIVATE=your-key-here

🧭 Windows PowerShell

$env:SERPER_KEY_PRIVATE="your-key-here"

---

🚀 How to Run VeriFastScore

Once installed, use the CLI tool:

verifastscore --input_file responses.jsonl

Or with custom arguments:

verifastscore \
  --input_file responses.jsonl \
  --data_dir ./data \
  --output_dir ./outputs \
  --model_name rishanthrajendhran/VeriFastScore \
  --search_res_num 10

📌 Script Arguments

Argument	Type	Default	Description
--input_file	str	required	Input file (.jsonl) in --data_dir.
--data_dir	str	./data	Directory for input files.
--output_dir	str	./data	Where to write outputs.
--cache_dir	str	./data/cache	Directory to store SERPER search cache.
--model_name	str	rishanthrajendhran/VeriFastScore	Hugging Face model name or local path. The default value is the path to the trained VeriFastScore model on HuggingFace. Do not change this path unless you want to test your own model.
--search_res_num	int	10	Evidence snippets per sentence.

---

📥 Input Format

The input must be a .jsonl file with the following structure:

{
  ["id": "Optional_string_ID"]
  "response": "The capital of France is Paris.",
  [...]
}

Place the file in the --data_dir directory.

A sample input file can be found here

---

📤 Output

• ./outputs/evidence_*.jsonl – Response with retrieved evidence
• ./outputs/model_output/decomposition_verification_*.jsonl – Claim-level factuality labels
• ./outputs/time/verifastscore_time_*.pkl – Timing breakdown

Example output:

"claim_verification_result": [
  {"claim": "Paris is the capital of France.", "verification_result": "supported"},
  {"claim": "France is in South America.", "verification_result": "unsupported"}
]

Console output will include average VeriFastScore, timing per instance, and per stage.

---

⚙️ Optional Setup Tools

To prepare the Python environment:

1. Clone the repository and create the conda environment:

   conda env create -f environment.yml
   conda activate verifastscore

2. Download spaCy English tokenizer:

   python3 -m spacy download en_core_web_sm

3. Install PyTorch (choose the version that matches your CUDA setup):

   pip install torch torchvision torchaudio

4. Install FlashAttention:

   pip install flash-attn --no-build-isolation

---

📖 Citation

@misc{rajendhran2025verifastscorespeedinglongformfactuality,
      title={VeriFastScore: Speeding up long-form factuality evaluation}, 
      author={Rishanth Rajendhran and Amir Zadeh and Matthew Sarte and Chuan Li and Mohit Iyyer},
      year={2025},
      eprint={2505.16973},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.16973}, 
}

---

📄 License

This project is licensed under the Apache 2.0 License.