Artefactual

Estimating how likely a language model answer is to be a hallucination.

Artefactual is a Python module that assigns a language model's answer a probability of being a hallucination. It reads the answer that has already been generated, together with the token probabilities returned alongside it, and needs nothing else from the model.

Quick start

from openai import OpenAI

from artefactual.scoring import wepr

MODEL = "mistralai/Ministral-8B-Instruct-2410"          # the model being scored
DETECTOR = "chicham/artefactual-wepr-ministral"        # the detector trained for it

client = OpenAI(base_url="https://your-provider.example/v1")  # any OpenAI-compatible endpoint
response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Who wrote the Rust book?"}],
    logprobs=True,
    top_logprobs=15,
)

detector = wepr(DETECTOR)
print(detector.predict_proba(response)[:, 1])   # P(hallucination) per sequence
print(detector.predict_token_proba(response))   # ...and per token

A detector is trained for one model, so DETECTOR must be the one published for whatever produced the response — see published detectors for the pairs. There is no default threshold; choosing one is covered in the user guide.

No endpoint is needed to try the library. The example notebooks run against two checked-in responses and need no GPU, API key or model download.

Requirements

Two conditions apply to whatever produced the response:

1. logprobs and top_logprobs are enabled. Providers that do not expose them cannot be scored at all.
2. top_logprobs is at least k, the rank count the detector's own weights were trained at. The shipped files use k = 15.

Neither needs auditing in advance — a response that fails one is refused by name. Why a narrow response is refused rather than padded is in the reference.

Installation

pip install artefactual

Dependencies

Artefactual requires:

• Python (>= 3.11)
• NumPy
• scikit-learn
• pydantic
• beartype

Two optional extras are available: [adapters] installs langfuse and openai for the integration examples, and [docs] installs Sphinx and the theme for building the documentation.

From source

git clone https://github.com/artefactory/artefactual
cd artefactual
uv sync

Usage

Two detectors are provided. wepr is the default and the more accurate; epr fits a single coefficient instead of 2k, for when labelled data is scarce. Both take the same arguments and return the same type.

from artefactual.scoring import epr, wepr

wepr("chicham/artefactual-wepr-phi4")         # a published detector, by its own repo
wepr("/path/to/my_detector.skops")           # one you trained yourself
wepr(k=15, trainable=True).fit(responses, y)  # fit your own, y is 0/1 per sequence
epr("chicham/artefactual-epr-phi4")           # the single-coefficient variant

Scoring a batch, reading per-token scores, scoring Langfuse traces, composing into GridSearchCV and training a detector for a model that is not shipped are covered in the user guide.

Published detectors

A detector is named by its own Hugging Face repository, not by the model it scores. Pick the row for the model that produced the responses, and the column for the reduction:

All are trained at k = 15. Both factories also accept a path to a .skops file, so a detector you trained yourself is named the same way one published here is — the package holds no list of models, and publishing another detector needs no release.

Limitations

• Not every provider can be scored. The requirements above rule out any provider that hides logprobs, and any that caps top_logprobs below the detector's k.
• Detectors are model-specific. Scoring a model that is not shipped requires training a detector for it, which requires labelled data.
• The probability is only as good as its training data. Rankings transfer more readily than absolute values.
• It measures uncertainty, not truth. A model that is confidently wrong is not uncertain, and scores low. This complements retrieval grounding or a judge rather than replacing either.
• Scoring is per sequence. There is no cross-response consistency check.

Results

ROC-AUC on TriviaQA hallucination detection at k = 15, as reported in Table 1 of the paper:

The full tables and the procedure that produced them are in the scripts/ecir subdirectory.

Documentation

• Documentation and user guide: https://artefactory.github.io/artefactual/
• Example notebooks: https://artefactory.github.io/artefactual/examples/
• How it works: https://artefactory.github.io/artefactual/guide/how-it-works.html

Development

Contributions are welcome — see CONTRIBUTING.md.

• Source code: https://github.com/artefactory/artefactual
• Issue tracker: https://github.com/artefactory/artefactual/issues

uv sync
uv run pytest tests        # test suite (uv sync resolves the env first)
uv run pytest tests --cov  # with coverage

uvx ruff check src tests   # lint — a standalone tool, no project env needed
uvx ruff format src tests
uvx --from shellcheck-py shellcheck scripts/ecir/*.sh

Versioning

Releases are dated (YYYY.MM.PATCH). Breaking changes to the public surface are called out in the release notes.

Citation

If artefactual is useful in your research, please cite our paper, accepted for publication at ECIR 2026:

@misc{moslonka2025learnedhallucinationdetectionblackbox,
      title={Learned Hallucination Detection in Black-Box LLMs using Token-level Entropy Production Rate},
      author={Charles Moslonka and Hicham Randrianarivo and Arthur Garnier and Emmanuel Malherbe},
      year={2025},
      eprint={2509.04492},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.04492},
}

License

MIT — no limitation of usage, including for commercial applications.