medeval-framework

A rigorous, open-source Python evaluation framework designed to benchmark medical Large Language Models (LLMs) for clinical accuracy, hallucination rates, model calibration, and safety.

📖 Table of Contents

• Key Features
• Installation
  • Install via Pip (Recommended)
  • Install from Source
• Quickstart
  • Command Line Interface (CLI)
  • Python Orchestration API
• Repository Structure
• Development & Testing
• License

🌟 Key Features

• Multi-Dataset Benchmarks: Out-of-the-box loaders for standardized medical datasets (MedQA, PubMedQA).
• Dual Clinical Safety Audit:
  • Deterministic Checker: Fast regex scanning for explicit clinical contraindications in Sickle Cell Disease and Cardiology.
  • Semantic Safety Net: NLI-based hazard verification (SemanticSafetyChecker) to flag context-dependent medical hazards.
• Unified Model & PEFT Connectors: Modular drivers to query API models (OpenAI) or execute local PyTorch/Transformers weights (Hugging Face) and PEFT/LoRA adapters smoothly.
• NLP & Semantic Accuracy Engines: Exact Match comparison and BERTScore semantic similarity scoring (SemanticSimilarityScorer).
• NLI Hallucination Detection: Cross-encoder Natural Language Inference (NLIHallucinationDetector) evaluating predictions (hypothesis) against authoritative clinical facts (ground_truth).
• Advanced Calibration Suite: Vectorized calculation of Expected Calibration Error (ECE), Maximum Calibration Error (MCE), and Brier Score.
• CLI & Report Generator: Command-line medeval interface and structured JSON report exporter for auditability.

⚙️ Installation

1. Install via Pip (Recommended)

medeval-framework is available on PyPI. You can install the core framework or include optional ML/NLP extras:

# Core installation (numpy-only, lightweight)
pip install medeval-framework

# Full ML & NLP stack (Transformers, PyTorch, evaluate, datasets, bert_score, peft)
pip install medeval-framework[nlp]

# Complete installation including development and testing tools
pip install medeval-framework[all]

2. Install from Source

For development or contributing:

# Clone the repository
git clone https://github.com/TeslaInch/medeval-framework.git
cd medeval-framework

# Install editable package with all extras
pip install -e ".[all]"

🚀 Quickstart

1. Command Line Interface (CLI)

Run evaluations directly from your terminal using the medeval command:

# Get full usage help and flag options
medeval --help

# Run evaluation on MedQA using OpenAI GPT-4o with Sickle Cell safety audit (default)
export OPENAI_API_KEY="your-api-key"
medeval --model gpt-4o --dataset medqa --limit 20 --output report.json

# Run evaluation on Hugging Face model or PEFT adapter with explicit safety selection
# Options for --safety: sickle_cell (default), none
medeval \
  --model "microsoft/Phi-3.5-mini-instruct" \
  --dataset medqa \
  --safety sickle_cell \
  --device "cuda:0" \
  --limit 50 \
  --output base_model_report.json

CLI Options Reference

2. Python Orchestration API

Create customized evaluation pipelines tailored to specific clinical domains. You can configure individual safety checkers (SickleCellSafetyChecker, CardiologySafetyChecker, SemanticSafetyChecker) or combine them into a composite SafetySuite:

from medeval.benchmark import BenchmarkLoader
from medeval.models.huggingface import HuggingFaceConnector
from medeval.runner import BenchmarkRunner
from medeval.safety import (
    SickleCellSafetyChecker,
    CardiologySafetyChecker,
    SemanticSafetyChecker,
    SafetySuite,
)
from medeval.report import export_report_to_json

# 1. Load benchmark dataset
loader = BenchmarkLoader(split="test", max_samples=10)
samples = loader.load_medqa()

# 2. Instantiate Model Connector (Hugging Face base or PEFT adapter)
model = HuggingFaceConnector(model_name="microsoft/Phi-3.5-mini-instruct", device="cuda:0")

# 3. Configure Safety Checker
# Option A: Domain-specific checker (e.g. Sickle Cell Disease only)
safety_checker = SickleCellSafetyChecker()

# Option B: Multi-domain SafetySuite (Sickle Cell + Cardiology + NLI Semantic Net)
# safety_suite = SafetySuite([
#     SickleCellSafetyChecker(),
#     CardiologySafetyChecker(),
#     SemanticSafetyChecker(device=0)
# ])

# 4. Initialize and execute BenchmarkRunner
runner = BenchmarkRunner(model=model, safety_checker=safety_checker, ignore_errors=True)
report = runner.run(samples)

# 5. Export structured JSON report
export_report_to_json(report, "evaluation_report.json")

📁 Repository Structure

medeval/
├── medeval/
│   ├── models/               # Model Connectors (Base, HF, PEFT, OpenAI, Mock)
│   ├── safety/               # Safety Checkers (SickleCell, Cardiology, Semantic, SafetySuite)
│   ├── accuracy.py           # Scorers (Exact Match, BERTScore F1)
│   ├── benchmark.py          # Benchmark Loaders (MedQA, PubMedQA)
│   ├── calibration.py        # Calibration Suite (ECE, MCE, Brier Score)
│   ├── hallucination.py      # NLI Cross-Encoder Hallucination Engine
│   ├── report.py             # Metric aggregation & JSON serialization
│   ├── runner.py             # BenchmarkRunner pipeline orchestrator
│   └── structures.py         # Data contracts (MedicalEvalSample & EvaluationReport)
├── tests/                    # 169 Unit & Integration Tests
├── pyproject.toml            # Ruff & Mypy configurations
├── setup.py                  # PyPI Packaging configuration
└── requirements.txt          # Package dependencies

🧪 Development & Testing

Ensure style alignment and type safety before submitting pull requests:

# Run full pytest test suite
pytest

# Run Ruff style & linting check
ruff check .

# Run Mypy static type verification
mypy medeval/

📄 License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.