openfactcheck

OpenFactCheck Logo

An Open-source Factuality Evaluation Demo for LLMs

Overview • Installation • Usage • HuggingFace Demo • Documentation

Overview

OpenFactCheck is an open-source repository designed to facilitate the evaluation and enhancement of factuality in responses generated by large language models (LLMs). This project aims to integrate various fact-checking tools into a unified framework and provide comprehensive evaluation pipelines, with built-in solvers for English and five additional languages (Arabic, Bulgarian, Chinese, Italian, and Urdu).

Supported Solvers

OpenFactCheck ships with several fact-checking pipelines you can use out of the box:

English

factool — pipeline from FacTool
factcheckgpt — pipeline from FactCheck-GPT
rarr — Retrieval-Augmented Research and Revision

Multilingual (Arabic, Bulgarian, Chinese, and Italian are new in v1.1.0)

arabicfactcheck — Arabic claim verification
bulgarianfactcheck — Bulgarian claim verification
chinesefactcheck — Chinese claim verification
italianfactcheck — Italian claim verification
urdufactcheck — Urdu claim verification

Each multilingual solver follows the same five-stage pattern: cp (claim processing) → rtv / rtv_tr / rtv_thtr (retrieval variants) → vfr (verification).

Utility

dummy — passthrough/no-op (testing)
tutorial — minimal example for building your own solver
webservice — wrap any HTTP API as a solver

Installation

You can install the package from PyPI using pip:

pip install openfactcheck

Usage

First, you need to initialize the OpenFactCheckConfig object and then the OpenFactCheck object.

from openfactcheck import OpenFactCheck, OpenFactCheckConfig

# Initialize the OpenFactCheck object
config = OpenFactCheckConfig()
ofc = OpenFactCheck(config)

Response Evaluation

You can evaluate a response using the ResponseEvaluator class.

# Evaluate a response
result = ofc.ResponseEvaluator.evaluate(response: str)

LLM Evaluation

We provide FactQA, a dataset of 6480 questions for evaluating LLMs. Onc you have the responses from the LLM, you can evaluate them using the LLMEvaluator class.

# Evaluate an LLM
result = ofc.LLMEvaluator.evaluate(model_name: str,
                                   input_path: str)

Checker Evaluation

We provide FactBench, a dataset of 4507 claims for evaluating fact-checkers. Once you have the responses from the fact-checker, you can evaluate them using the CheckerEvaluator class.

# Evaluate a fact-checker
result = ofc.CheckerEvaluator.evaluate(checker_name: str,
                                       input_path: str)

Cite

If you use OpenFactCheck in your research, please cite the following:

@article{wang2024openfactcheck,
  title        = {OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs},
  author       = {Wang, Yuxia and Wang, Minghan and Iqbal, Hasan and Georgiev, Georgi and Geng, Jiahui and Nakov, Preslav},
  journal      = {arXiv preprint arXiv:2405.05583},
  year         = {2024}
}

@article{iqbal2024openfactcheck,
  title        = {OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs},
  author       = {Iqbal, Hasan and Wang, Yuxia and Wang, Minghan and Georgiev, Georgi and Geng, Jiahui and Gurevych, Iryna and Nakov, Preslav},
  journal      = {arXiv preprint arXiv:2408.11832},
  year         = {2024}
}

@software{hasan_iqbal_2024_13358665,
  author       = {Hasan Iqbal},
  title        = {hasaniqbal777/OpenFactCheck: v1.1.0},
  month        = {aug},
  year         = {2024},
  publisher    = {Zenodo},
  version      = {v1.1.0},
  doi          = {10.5281/zenodo.13358665},
  url          = {https://doi.org/10.5281/zenodo.13358665}
}

Release files for openfactcheck 1.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for openfactcheck 1.1.2
File	Size	Uploaded
openfactcheck-1.1.2.tar.gz	6.1 MB	Apr 28, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for openfactcheck 1.1.2
File	Interpreter	ABI	Platform	Reset
openfactcheck-1.1.2-py3-none-any.whl 6.2 MB Apr 28, 2026	Python 3	none	any	Details

Total release size: 12.3 MB

Release files / openfactcheck-1.1.2.tar.gz

Download URL	openfactcheck-1.1.2.tar.gz
Size	6.1 MB
Tags	Source
SHA-256 checksum How to use checksums	`e17a73b6a233648ad39310aa8a464e419cdef9d174fa56def9c795090d4e6c92`
BLAKE2b-256 checksum How to use checksums	`f373abb04f77963e0198ebd04b9924020b354e65a2b410f58004f9f09e4e3e33`
Upload date	Apr 28, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.11.15`

Release files / openfactcheck-1.1.2-py3-none-any.whl

Download URL	openfactcheck-1.1.2-py3-none-any.whl
Size	6.2 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`2b00259e9bcf4db02a05542e68da7e70e450fdd7862ce0f032395f72dfd942e5`
BLAKE2b-256 checksum How to use checksums	`bd426cee61f0bd6dd7ca74cad4e9584026693eb77eddf55926f1db5629e88931`
Upload date	Apr 28, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.11.15`