epb-benchmark 1.0.0

Epistemic Pathology Benchmark - The MLPerf of AI Truth Systems

EPB: Epistemic Pathology Benchmark

The MLPerf of AI Truth Systems

EPB (Epistemic Pathology Benchmark) is a comprehensive benchmark for evaluating epistemic integrity in AI systems. It measures four critical pathologies that affect AI truthfulness and reliability:

• Mirror Loop: Collapse in recursive self-refinement
• Confabulation: Fabrication and persistence of false information
• Violation State: Refusal contamination of benign prompts
• Echo Chamber: Synthetic drift and self-reinforcement

Quick Start

Installation

pip install epb-benchmark

Or install from source:

git clone https://github.com/Course-Correct-Labs/epb-benchmark.git
cd epb-benchmark
pip install -e .

Running the Benchmark

1. Initialize a configuration file:

epb init-config

2. Edit epb_config.yaml to set your model and API key:

adapter:
  provider: "openai"  # or "anthropic"
  model_name: "gpt-4"
  api_key_env: "OPENAI_API_KEY"

3. Set your API key:

export OPENAI_API_KEY="your-api-key-here"

4. Run the benchmark:

epb run --config epb_config.yaml

5. Score the results:

epb score --run-dir runs/YYYYMMDD_HHMMSS

What EPB Measures

EPB evaluates four distinct pathologies, each with an explicit metric:

1. Mirror Loop (EPB Phi)

Measures stability in recursive self-refinement. Models are asked to iteratively critique and improve their own outputs. Collapse occurs when the model gets stuck in repetitive patterns.

Score: 0-100 (higher is better)

2. Confabulation (EPB Persistence)

Measures fabrication of false information and its persistence after challenge. Models are asked unanswerable questions, then challenged on their answers.

Score: 0-100 (higher is better, less persistent confabulation)

3. Violation State (EPB Contamination)

Measures refusal contamination after seeing disallowed content. Models receive a violation request (which should be refused), followed by benign requests.

Score: 0-100 (higher is better, less contamination)

4. Echo Chamber (EPB Drift)

Measures semantic drift through iterative summarization. Models repeatedly summarize their own outputs, and drift is measured using TF-IDF cosine similarity.

Score: 0-100 (higher is better, less drift)

Overall Score: EPB Truth

The overall EPB Truth score is a weighted average of the four sub-scores (default: equal weighting).

Certification Levels:

• Platinum: 95+
• Gold: 85+
• Silver: 70+
• Bronze: 50+

EPB v1 Test Suite

EPB v1 includes:

• 20 Mirror Loop prompts
• 30 Confabulation questions
• 10 Violation State sequences
• 10 Echo Chamber scenarios

Total: 70 test tasks designed for quality over quantity.

Documentation

• Quickstart Guide
• Methodology
• Scoring Details
• API Reference

Leaderboard

Submit your results to the public leaderboard:

export EPB_LEADERBOARD_URL="https://epb.coursecorrect.org/api"
export EPB_API_KEY="your-leaderboard-api-key"
epb submit --results runs/YYYYMMDD_HHMMSS/results.json

View the leaderboard at: https://epb.coursecorrect.org

Architecture

EPB is designed to be:

• Model-agnostic: Works with any LLM through simple adapters
• Reproducible: Explicit metrics and deterministic scoring
• Extensible: Easy to add new batteries and adapters
• Transparent: Open-source specifications and scoring code

Supported Models

Out of the box, EPB supports:

• OpenAI GPT models (GPT-4, GPT-3.5, etc.)
• Anthropic Claude models

To add support for other models, implement the ModelClient interface in epb/adapters/.

Citation

If you use EPB in your research, please cite:

@software{epb2025,
  title = {EPB: Epistemic Pathology Benchmark},
  author = {Course Correct Labs},
  year = {2025},
  url = {https://github.com/Course-Correct-Labs/epb-benchmark}
}

Contributing

We welcome contributions! Please see our contributing guidelines.

Areas for contribution:

• New model adapters
• Additional test tasks
• Improved scoring heuristics
• Bug fixes and documentation

License

MIT License - see LICENSE for details.

About

EPB is developed by Course Correct Labs, a research organization focused on epistemic integrity in AI systems.

Related work:

• Mirror Loop
• Recursive Confabulation
• Violation State
• Echo Chamber Zero

Support

• GitHub Issues: Report bugs or request features
• Documentation: docs/
• Contact: [e-mail]