verifastscore 0.1.3

Fast, end-to-end factuality evaluation for long-form LLM responses.

🔍⚡ VeriFastScore

VeriFastScore is a fast and efficient factuality evaluation tool that jointly extracts and verifies fine-grained factual claims from long-form LLM-generated responses, conditioned on retrieved web evidence.

This repository packages VeriFastScore as a pip-installable Python package with a command-line interface (verifastscore), simplifying usage and deployment.

---

HuggingFace Links

• Model
• Dataset

---

📦 Installation

VeriFastScore supports both CPU and GPU setups. Please follow the appropriate instructions below:

▶️ CPU Installation [Default]

To install VeriFastScore for CPU-only usage:

pip install verifastscore
python3 -m spacy download en_core_web_sm

▶️ GPU Installation (with FlashAttention) [Recommended]

If you want GPU acceleration with FlashAttention, you must install compatible versions of torch and flash-attn manually before installing verifastscore.

# Install CUDA-enabled PyTorch (adjust for your CUDA version)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

# Install flash-attn (manually, requires build tools and matching CUDA)
pip install flash-attn --no-build-isolation

# Then install verifastscore
pip install verifastscore

# Finally, download the spaCy model
python3 -m spacy download en_core_web_sm

▶️ Development install (local)

git clone https://github.com/rishanthrajendhran/verifastscore.git
cd verifastscore
pip install -e .
python3 -m spacy download en_core_web_sm

---

🔐 Set Your SERPER API Key

VeriFastScore retrieves external evidence using Serper.dev. You'll need to set your API key in the environment:

💻 Linux/macOS

export SERPER_KEY_PRIVATE="your-key-here"

Add to ~/.bashrc or ~/.zshrc for permanence:

echo 'export SERPER_KEY_PRIVATE="your-key-here"' >> ~/.bashrc
source ~/.bashrc

🪟 Windows CMD

set SERPER_KEY_PRIVATE=your-key-here

🧭 Windows PowerShell

$env:SERPER_KEY_PRIVATE="your-key-here"

---

🚀 How to Run VeriFastScore

Once installed, use the CLI tool:

verifastscore --input_file responses.jsonl

Or with custom arguments:

verifastscore \
  --input_file responses.jsonl \
  --data_dir ./data \
  --output_dir ./outputs \
  --model_name rishanthrajendhran/VeriFastScore \
  --search_res_num 10

📌 Script Arguments

Argument	Type	Default	Description
--input_file	str	required	Input file (.jsonl) in --data_dir.
--data_dir	str	./data	Directory for input files.
--output_dir	str	./data	Where to write outputs.
--cache_dir	str	./data/cache	Directory to store SERPER search cache.
--model_name	str	rishanthrajendhran/VeriFastScore	Hugging Face model name or local path.
--search_res_num	int	10	Evidence snippets per sentence.

---

📥 Input Format

The input must be a .jsonl file with the following structure:

{
  "question" [Optional]: "What is the capital of France?",
  "response": "The capital of France is Paris.",
  "prompt_source"  [Optional]: "tulu", 
  "model" [Optional]: "gpt-4o"
}

Place the file in the --data_dir directory.

---

📤 Output

• ./outputs/evidence_*.jsonl – Response with retrieved evidence
• ./outputs/model_output/decomposition_verification_*.jsonl – Claim-level factuality labels
• ./outputs/time/verifastscore_time_*.pkl – Timing breakdown

Example output:

"claim_verification_result": [
  {"claim": "Paris is the capital of France.", "verification_result": "supported"},
  {"claim": "France is in South America.", "verification_result": "unsupported"}
]

Console output will include average VeriFastScore, timing per instance, and per stage.

---

⚙️ Optional Setup Tools

To prepare the Python environment:

1. Clone the repository and create the conda environment:

   conda env create -f environment.yml
   conda activate verifastscore

2. Download spaCy English tokenizer:

   python3 -m spacy download en_core_web_sm

3. Install PyTorch (choose the version that matches your CUDA setup):

   pip install torch torchvision torchaudio

4. Install FlashAttention:

   pip install flash-attn --no-build-isolation

---

📖 Citation

@misc{rajendhran2025verifastscorespeedinglongformfactuality,
      title={VeriFastScore: Speeding up long-form factuality evaluation}, 
      author={Rishanth Rajendhran and Amir Zadeh and Matthew Sarte and Chuan Li and Mohit Iyyer},
      year={2025},
      eprint={2505.16973},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.16973}, 
}

---

📄 License

This project is licensed under the Apache 2.0 License.