truthscore-llm 0.1.0

A research library for evaluating truthfulness of LLM outputs based on evidence agreement, self-consistency, and retrieval coverage

TruthScore-LLM

A research-oriented Python library for evaluating the truthfulness of LLM-generated outputs based on evidence agreement, self-consistency, and retrieval coverage.

Overview

TruthScore-LLM implements a multi-dimensional scoring system that assesses the reliability of answers generated by large language models. The library evaluates answers across four key dimensions:

• Evidence Agreement: How well retrieved evidence documents support the answer (entailment checking)
• Self-Consistency: Internal coherence and logical consistency of the answer
• Retrieval Coverage: Comprehensiveness of supporting evidence
• Language Confidence: Linguistic quality and certainty indicators

These component scores are aggregated into a single truth score (0.0 to 1.0) and a categorical decision (ACCEPT, QUALIFIED, or REFUSE).

Installation

Development Installation

To install the library in development mode:

git clone <repository-url>
cd truthscore
pip install -e .

Future PyPI Installation

Once published, the library will be installable via:

pip install truthscore-llm

Quick Start

from truthscore import TruthScorer

# Initialize scorer
scorer = TruthScorer()

# Evaluate an answer
result = scorer.score(
    question="Does vitamin C prevent the common cold?",
    answer="Vitamin C prevents the common cold."
)

# Access results
print(f"Truth Score: {result['truth_score']:.3f}")
print(f"Decision: {result['decision']}")
print(f"Evidence Score: {result['evidence_score']:.3f}")
print(f"Consistency: {result['consistency']:.3f}")
print(f"Language Confidence: {result['language_confidence']:.3f}")
print(f"Coverage: {result['coverage']:.3f}")

Output Format

The score() method returns a dictionary with the following structure:

{
    "truth_score": float,          # Overall truth score [0.0, 1.0]
    "decision": str,               # "ACCEPT" | "QUALIFIED" | "REFUSE"
    "evidence_score": float,       # Evidence agreement [0.0, 1.0]
    "consistency": float,          # Self-consistency [0.0, 1.0]
    "language_confidence": float,  # Language confidence [0.0, 1.0]
    "coverage": float              # Retrieval coverage [0.0, 1.0]
}

Configuration

You can customize scoring behavior by providing a custom configuration:

from truthscore import TruthScorer, TruthScoreConfig

# Create custom configuration
config = TruthScoreConfig(
    evidence_weight=0.5,
    consistency_weight=0.3,
    coverage_weight=0.15,
    language_weight=0.05,
    accept_threshold=0.80,
    qualified_threshold=0.60
)

# Initialize scorer with custom config
scorer = TruthScorer(config=config)

Project Structure

truthscore/
├── truthscore/           # Main package
│   ├── __init__.py      # Package initialization and exports
│   ├── score.py         # TruthScorer main class
│   ├── config.py        # Configuration management
│   ├── retrieve.py      # Evidence retrieval module
│   ├── nli.py           # Natural Language Inference module
│   ├── consistency.py   # Consistency evaluation module
│   └── coverage.py      # Coverage evaluation module
├── examples/            # Usage examples
│   └── example.py
├── tests/               # Unit tests
│   └── test_score.py
├── README.md            # This file
├── pyproject.toml       # Package metadata
└── setup.cfg            # Setuptools configuration

Running Tests

python -m pytest tests/

Or using unittest:

python -m unittest tests.test_score

Running Examples

python examples/example.py

Research Disclaimer

Important: This library is provided for research purposes. The scoring mechanisms implemented here are experimental and based on placeholder implementations for key components (retrieval, NLI, consistency checking).

• The current implementations use simple heuristics and are designed to be deterministic for testing purposes.
• Do not use this library as the sole basis for critical decisions without:
  • Validating results against domain-specific ground truth
  • Replacing placeholder implementations with production-grade systems
  • Calibrating thresholds for your specific use case
  • Conducting thorough evaluation and error analysis

The library is structured to facilitate easy replacement of placeholder components with real systems (e.g., trained NLI models, vector databases for retrieval, consistency checking systems).

Contributing

Contributions are welcome! Please ensure that:

• Code follows the existing style and structure
• All tests pass
• New features include appropriate tests
• Documentation is updated

License

MIT License

Citation

If you use this library in your research, please cite:

@software{truthscore2024,
  title={TruthScore-LLM: A Research Library for Evaluating Truthfulness of LLM Outputs},
  author={TruthScore Contributors},
  year={2024},
  url={https://github.com/yourusername/truthscore-llm}
}