Factly is a modern CLI tool designed to evaluate the factuality of Large Language Models (LLMs) on the MMLU (Massive Multitask Language Understanding) benchmark. It provides a robust framework for prompt engineering experiments and factual accuracy assessment.
Features
Evaluate LLM factuality on the MMLU benchmark with detailed results
Support for various prompt engineering experiments via configurable system instructions
Generate comparative visualizations of factuality scores across models and prompts
Structured output for easy analysis and comparison
Built with modern Python tooling (Python 3.12, uv, click, pydantic)
Extensible and reproducible evaluation workflows
Quick Start
# Run factuality evaluation with default settings
factly evaluate
# Run evaluation and generate plots
factly evaluate --plot
# Get help on all available options
factly evaluate --helpThat’s it! The tool uses optimized default parameters and saves all outputs to the output directory.
For more advanced usage, including saving results and evaluation, see the Usage Guide.
Project Information
Factly is released under the MIT License, its documentation lives at Read the Docs, the code on GitHub, and the latest release on PyPI. It’s rigorously tested on Python 3.12+.
If you’d like to contribute to Factly you’re most welcome!
Support
Should you have any question, any remark, or if you find a bug, or if there is something you can’t do with the Factly, please open an issue.
1 maintainer
Metadata
1 maintainer
Release files for factly-eval 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| factly_eval-1.0.1.tar.gz | 148.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| factly_eval-1.0.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 167.9 kB
Release files / factly_eval-1.0.1.tar.gz
| Download URL | factly_eval-1.0.1.tar.gz |
|---|---|
| Size | 148.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fd8ab44dad4e380adb596335e5a04904113e6c85a74f978bffef337f7cf1fcec
|
|
BLAKE2b-256 checksum How to use checksums |
616fb58dd73e64039f713b49f2197a85fdf9626faf0a0f2dab88b4a86be014f5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 30, 2025.
Transparency logRelease files / factly_eval-1.0.1-py3-none-any.whl
| Download URL | factly_eval-1.0.1-py3-none-any.whl |
|---|---|
| Size | 19.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
46e9f380b387c40a3aacf666b4645707448589bb061e01acc04b00af5ca22d02
|
|
BLAKE2b-256 checksum How to use checksums |
a82484304b0e7eb012541031b4187668157510d31b95bbb3ce7ed2f16238ed4f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 30, 2025.
Transparency log
