medeval-framework

A rigorous, open-source Python evaluation framework designed to benchmark medical Large Language Models (LLMs) and PEFT/LoRA adapters for clinical accuracy, hallucination rates, model calibration, and safety.

📖 Table of Contents

• Key Features
• Installation
  • Install via Pip (Recommended)
  • Install from Source
• Quickstart
  • Command Line Interface (CLI)
  • Python Orchestration API
  • Multi-Model Side-by-Side Comparison
• Repository Structure
• Development & Testing
• License

🌟 Key Features

• Multi-Dataset Benchmarks: Out-of-the-box loaders for standard medical datasets (MedQA, PubMedQA, MedMCQA, MMLU-Medical).
• Custom Proprietary Data: Effortlessly evaluate your own internal datasets via Generic JSONL Loading (air-gapped PHI-safe) or dynamic HuggingFace Dataset ingestion.
• Dynamic Topic Filtering: Filter any benchmark dataset by medical domain or disease keyword on the fly (e.g. --topic "sickle", --topic "cardiac", --topic "renal").
• Universal SOTA Connectors: Query local PyTorch weights, PEFT/LoRA adapters (with automatic base-model merging), local RAG architectures (llama_cpp / ChromaDB), OpenAI APIs (gpt-4o), or SOTA routers (OpenRouter) driving Claude 3.5 Sonnet, Llama 3.1 70B/405B, DeepSeek, and Gemini.
• Dual Clinical Safety Audit:
  • Deterministic Checker: Pure-Python regex engine detecting explicit contraindications in Sickle Cell Disease and Cardiology.
  • Semantic Safety Net: Cross-encoder NLI hazard verification (SemanticSafetyChecker) flagging context-dependent medical risks.
• Context-Injected Hallucination Detection: Cross-encoder Natural Language Inference (NLIHallucinationDetector) evaluating prediction hypotheses against ground-truth clinical facts with full context injection to minimize false positives.
• Advanced Calibration & Metrics Suite:
  • Vectorized Expected Calibration Error (ECE), Maximum Calibration Error (MCE), and Brier Score.
  • Generative Sequence Mean: Calculates robust confidence (y_prob) by averaging generated token probabilities, natively supporting OpenAI logprobs.
  • Zero-Shot CoT Uncertainty Extraction: Calculates probabilities directly from <think> reasoning traces for models that do not output logprobs (e.g. DeepSeek R1, OpenAI o1).
  • Dynamic Bootstrap Resampling: Automatically calculates rigorous 95% Confidence Intervals for Accuracy and Hallucination Rates.
• Incremental Checkpointing: Instantly saves results to a .jsonl stream, allowing you to resume interrupted evaluations flawlessly without re-running API calls.
• Robust Answer Extraction: High-fidelity regex strategies that flawlessly strip markdown and extract multi-choice keys from verbose clinical reasoning.
• Multi-Format Export & Comparison: Export structured reports to JSON, Markdown tables (.md), or interactive HTML dashboards (.html), with built-in side-by-side model comparison tools (compare_reports).

📊 Public SOTA Evaluation Benchmarks

The following side-by-side evaluation metrics were generated directly by the medeval-framework against 60 highly-filtered clinical queries from MedQA and PubMedQA.

Note: High Accuracy combined with high Hallucination Rates points towards memorization of open-source datasets over true clinical reasoning.

⚙️ Installation

1. Install via Pip (Recommended)

medeval-framework is available on PyPI:

# Core lightweight installation (NumPy & pure-Python engines)
pip install medeval-framework

# Full ML & NLP stack (Transformers, PyTorch, Datasets, PEFT, BERTScore)
pip install medeval-framework[nlp]

# Complete installation including development and test suites
pip install medeval-framework[all]

2. Install from Source

For development or contributing:

git clone https://github.com/TeslaInch/medeval-framework.git
cd medeval-framework
pip install -e ".[all]"

🚀 Quickstart

1. Command Line Interface (CLI)

Run evaluations directly from your terminal:

# Check version and available flags
medeval --version
medeval --help

# Evaluate OpenAI GPT-4o on MedQA
export OPENAI_API_KEY="your-api-key"
medeval --model gpt-4o --dataset medqa --limit 20 --output report.json

# Evaluate Claude 3.5 Sonnet or Llama 3.1 via AgentRouter with custom API flags
medeval \
  --model "agentrouter:claude-3-5-sonnet" \
  --api-key "your_agentrouter_key" \
  --api-base-url "https://agentrouter.ai/v1" \
  --dataset medqa \
  --output report_claude.html

# Evaluate a specialized PEFT adapter filtered specifically for Sickle Cell questions
medeval \
  --model "TeslaInch/scd-phi35-adapter-v8" \
  --dataset medqa \
  --topic "sickle" \
  --output sickle_cell_report.md

CLI Options Reference

2. Python Orchestration API

Build custom evaluation pipelines in Python:

from medeval.benchmark import BenchmarkLoader
from medeval.models.openai_connector import OpenAIConnector
from medeval.runner import BenchmarkRunner
from medeval.safety import SickleCellSafetyChecker
from medeval.hallucination import NLIHallucinationDetector
from medeval.scorers import ExactMatchScorer, SemanticSimilarityScorer
from medeval.report import export_report_to_html, export_report_to_markdown

# 1. Load dataset with topic filtering (e.g. Cardiology questions)
loader = BenchmarkLoader(split="test", max_samples=20, topic="cardiac")
samples = loader.load_medqa()

# 2. Instantiate Model Connector (API Router or HuggingFace model)
model = OpenAIConnector(model_name="claude-3-5-sonnet", base_url="https://agentrouter.ai/v1")

# 3. Configure Metrics, Safety Checkers & Checkpointing
scorers = [
    ExactMatchScorer(),
    SemanticSimilarityScorer(model_name="microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract"),
]
hallucination_detector = NLIHallucinationDetector(model_name="pritamdeka/PubMedBERT-MNLI-MedNLI")

runner = BenchmarkRunner(
    model=model,
    scorers=scorers,
    safety_checker=SickleCellSafetyChecker(),
    hallucination_detector=hallucination_detector,
    checkpoint_dir="./checkpoints",  # Enables instant resume capability!
    ignore_errors=True,
)

# 4. Execute Benchmark
report = runner.run(samples)

# 5. Export report as styled Markdown or HTML dashboard
export_report_to_markdown(report, "cardiology_report.md")
export_report_to_html(report, "cardiology_report.html")

3. Multi-Model Side-by-Side Comparison

Compare base models, fine-tuned adapters, and SOTA models in Python:

from medeval.comparison import export_comparison_to_markdown, load_reports_from_files

# Load evaluation reports
reports = load_reports_from_files(
    ["base_model_report.json", "sickle_cell_adapter_report.json", "sota_claude_report.json"]
)

# Export side-by-side comparative Markdown matrix
export_comparison_to_markdown(reports, "model_comparison_matrix.md")

📁 Repository Structure

medeval/
├── medeval/
│   ├── models/               # Model Connectors (HF, PEFT, OpenAI, AgentRouter, Mock)
│   ├── safety/               # Safety Checkers (SickleCell, Cardiology, Semantic, SafetySuite)
│   ├── accuracy.py           # Scorers (Exact Match, BERTScore F1)
│   ├── benchmark.py          # Loaders (MedQA, PubMedQA, MedMCQA, MMLU-Medical)
│   ├── calibration.py        # Calibration Suite (ECE, MCE, Brier Score)
│   ├── comparison.py         # Multi-Model Side-by-Side Comparison Engine
│   ├── hallucination.py      # NLI Cross-Encoder Hallucination Engine
│   ├── report.py             # Metric aggregation (JSON, Markdown, HTML exporters)
│   ├── runner.py             # BenchmarkRunner pipeline orchestrator
│   └── structures.py         # Data contracts (MedicalEvalSample & EvaluationReport)
├── tests/                    # 46 Unit & Integration Tests
├── pyproject.toml            # Ruff & Mypy configurations
├── setup.py                  # PyPI Packaging configuration
└── requirements.txt          # Package dependencies

🧪 Development & Testing

Ensure style alignment and type safety before submitting pull requests:

# Run full pytest test suite
pytest

# Run Ruff style & linting check
ruff check .

# Run Mypy static type verification
mypy medeval/

📄 License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.