HeLLMholtz LLM Suite

A comprehensive Python package for unified LLM access, benchmarking, evaluation, and reporting. Built on top of aisuite with specialized support for Helmholtz Blablador models.

Features

• Unified Client: Single interface for OpenAI, Google, Anthropic, Ollama, and Helmholtz Blablador models
• Centralized Configuration: Environment-based configuration for all your projects
• Advanced Benchmarking: Compare model performance across temperatures, replications, and prompt categories
• LLM-as-a-Judge Evaluation: Automated evaluation with comprehensive statistical analysis
• Interactive Reports: HTML reports with Chart.js visualizations and Markdown summaries
• Flexible Prompt System: Support for both simple text files and structured JSON prompt collections
• Model Monitoring: Track Blablador model availability and configuration consistency
• PII Anonymization: Document anonymization with the ShinrAI PII 1.3 model, including leak validation, cross-chunk consistency, and prompt-variant benchmarking
• LM Evaluation Harness: Integration with EleutherAI's comprehensive evaluation suite
• LiteLLM Proxy: Built-in proxy server for model routing and load balancing
• Throughput Testing: Performance benchmarking for high-throughput scenarios
• Model Discovery: Dynamic model listing and availability checking (19+ BLABLADOR models currently available)

Installation

Basic Installation

pip install hellmholtz

Development Installation

For development with all optional dependencies:

git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz
pip install -e ".[eval,proxy]"

Poetry Installation

poetry install --with eval,proxy

Configuration

HeLLMholtz supports both project-level and user-level configuration.

Project-Level Configuration (.env)

1. Copy the example environment file:

cp .env.example .env

2. Configure your API keys in .env:

# OpenAI
OPENAI_API_KEY=your_openai_key

# Anthropic
ANTHROPIC_API_KEY=your_anthropic_key

# Google
GOOGLE_API_KEY=your_google_key

# Helmholtz Blablador
BLABLADOR_API_KEY=your_blablador_key
BLABLADOR_API_BASE=https://your-blablador-instance.com

# Optional: Default models
AISUITE_DEFAULT_MODELS='{"openai": "gpt-4o", "anthropic": "claude-3-haiku"}'

User-Level Configuration (Recommended for pipx users)

For persistent configuration across all projects, use the hellm-setup tool:

# Configure once (works globally)
hellm-setup --api-key "your-blablador-key" \
            --base-url "https://api.blablador.ai/v1" \
            --default-models "meta-llama/Meta-Llama-3.1-8B-Instruct,gpt-4"

Configuration Location: ~/.config/hellmholtz/.env

This is automatically loaded for all hellm commands, regardless of your current directory.

Configuration Precedence: Project .env overrides user config (useful for per-project overrides).

See Configuration Guide for more details.

Usage

Python API

Basic Chat Interface

from hellmholtz.client import chat

# Simple chat
response = chat("openai:gpt-4o", "Hello, how are you?")
print(response)

# With conversation history
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."},
]
response = chat("anthropic:claude-3-sonnet", messages)

Benchmarking

from hellmholtz.benchmark import run_benchmarks
from hellmholtz.core.prompts import load_prompts

# Load prompts from JSON file
prompts = load_prompts("prompts.json", category="reasoning")

# Run benchmarks
results = run_benchmarks(
    models=["openai:gpt-4o", "anthropic:claude-3-haiku", "blablador:gpt-4o"],
    prompts=prompts,
    temperatures=[0.1, 0.7, 1.0],
    replications=3,
)

# Analyze results
from hellmholtz.evaluation_analysis import EvaluationAnalyzer

analyzer = EvaluationAnalyzer()
analysis = analyzer.analyze_evaluation_results("results/benchmark_latest.json")
analyzer.print_analysis_summary(analysis)

PII Anonymization

Anonymize documents with the ShinrAI PII 1.3 model (blablador:alias-anonymizer by default). Results include the anonymized text, the detected PII entities, and a leak validation report.

from hellmholtz.anonymizer import anonymize_file, anonymize_text

# Anonymize an in-memory text
result = anonymize_text("Patient John Doe was admitted on 12.03.2025.")
print(result.text)  # anonymized text
print(result.entities)  # PII entities that were replaced
print(result.validation.summary)  # leak validation summary

# Anonymize a file; writes <stem>.anonymized<suffix> plus a JSON sidecar
# (entities, mapping, validation) next to the output
result = anonymize_file("notes/patient.md", write=True)

# The entity mapping is reversible for authorized re-identification
restored = result.restore()

Longer documents are automatically split into word-safe chunks and reassembled with a consistent replacement mapping across chunk boundaries. Prompt variants (default, framed, strict, multilingual, footer_off) can be selected via the prompt keyword.

Command Line Interface

HeLLMholtz provides a comprehensive CLI for all operations:

Chat Interface

# Simple chat
hellm chat --model openai:gpt-4o "Explain the theory of relativity"

# Interactive mode
hellm chat --model anthropic:claude-3-sonnet --interactive

# With system prompt
hellm chat --model blablador:gpt-4o --system "You are a coding assistant" "Write a Python function to calculate fibonacci numbers"

Benchmarking

# Basic benchmark
hellm bench --models openai:gpt-4o,anthropic:claude-3-haiku --prompts-file prompts.txt

# Advanced benchmark with evaluation
hellm bench \
  --models openai:gpt-4o,blablador:gpt-4o \
  --prompts-file prompts.json \
  --prompts-category reasoning \
  --temperatures 0.1,0.7,1.0 \
  --replications 3 \
  --evaluate-with openai:gpt-4o \
  --results-dir results/

# Throughput testing
hellm bench-throughput \
  --model openai:gpt-4o \
  --requests 100 \
  --concurrency 10 \
  --prompt "Write a short story about AI"

Evaluation and Analysis

# Analyze benchmark results
hellm analyze results/benchmark_latest.json --html-report analysis_report.html

# Generate reports
hellm report --results-file results/benchmark_latest.json --output report.md

Model Management

# List available Blablador models
hellm models

# Monitor model availability and test accessibility
hellm monitor --test-accessibility

# Check model configuration consistency
hellm monitor --check-config

PII Anonymization

Anonymize documents with the ShinrAI PII 1.3 model (default: blablador:alias-anonymizer, override with --model). Leak validation and fuzzy matching run by default; disable with --no-validate / --no-fuzzy.

# Anonymize a file and print the anonymized text to stdout
hellm anonymize notes/patient.md

# Write the anonymized text next to the input (with JSON sidecar)
hellm anonymize notes/patient.md --write

# Read from stdin and print the detected entity table
cat notes/patient.md | hellm anonymize --show-entities

# Machine-readable JSON output
hellm anonymize notes/patient.md --json > patient_anonymized.json

# Compare all prompt variants across documents, scored against ground truth
hellm anonymize-benchmark \
  tests/anonymizer/fixtures/sample_short_en.md \
  tests/anonymizer/fixtures/sample_german.md \
  --ground-truth tests/anonymizer/fixtures/ground_truth.json

# Benchmark specific variants with more iterations; reports land in reports/anonymizer/
hellm anonymize-benchmark docs/ -p framed,strict --iterations 3

Benchmark reports include per-variant latency (p50/p95), throughput, validation pass rate, entity recall, and leak rate, written as Markdown and JSON to reports/anonymizer/.

Weekly Automated Benchmarking

The repository includes a GitHub Actions workflow that automatically runs benchmarks weekly and updates reports:

• Scheduled: Runs every Sunday at 00:00 UTC
• Model Discovery: Automatically fetches latest Blablador models
• Performance Charts: Generates visual charts comparing model performance
• Multiple Formats: Creates HTML, Markdown, and PNG chart reports
• Auto-commit: Updates reports in the repository for public viewing

To enable automated benchmarking:

1. Set repository secrets for API keys:

   • BLABLADOR_API_KEY: Your Blablador API key
   • BLABLADOR_API_BASE: Blablador API base URL (optional)

2. The workflow will automatically:

   • Run benchmarks on selected models
   • Generate performance reports
   • Create visual charts
   • Commit updated reports to the repository

Reports are available in the reports/ directory and include:

• weekly_benchmark_report.html: Interactive HTML report
• weekly_benchmark_report.md: Markdown summary
• weekly_benchmark_chart.png: Performance visualization

Advanced Features

# Run LM Evaluation Harness
hellm lm-eval \
  --model openai:gpt-4o \
  --tasks hellaswag,winogrande \
  --limit 100

# Start LiteLLM proxy server
hellm proxy \
  --config litellm_config.yaml \
  --port 8000

Project Structure

hellmholtz/
├── cli.py                 # Command-line interface
├── client.py              # Unified LLM client
├── monitoring.py          # Model availability monitoring
├── evaluation_analysis.py # Statistical analysis and reporting
├── export.py              # Result export utilities
├── core/
│   ├── config.py          # Configuration management
│   └── prompts.py         # Prompt loading and validation
├── benchmark/
│   ├── runner.py          # Benchmark execution
│   ├── evaluator.py       # LLM-as-a-Judge evaluation
│   └── prompts.py         # Benchmark-specific prompts
├── providers/
│   ├── blablador_provider.py # Custom Blablador provider
│   ├── blablador_config.py   # Blablador model configuration
│   ├── blablador.py          # Blablador utilities
│   └── __init__.py
├── reporting/
│   ├── html.py            # HTML report generation
│   ├── markdown.py        # Markdown report generation
│   ├── stats.py           # Statistical calculations
│   ├── utils.py           # Reporting utilities
│   └── templates/         # HTML templates
└── integrations/
    ├── lm_eval.py         # LM Evaluation Harness integration
    └── litellm.py         # LiteLLM proxy integration

Prompt System

HeLLMholtz supports two prompt formats:

Simple Text Format (prompts.txt)

What is the capital of France?
Explain quantum computing in simple terms.
Write a Python function to reverse a string.

Structured JSON Format (prompts.json)

[
  {
    "id": "capital-france",
    "category": "knowledge",
    "description": "Test basic geographical knowledge",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "expected_output": "Paris"
  },
  {
    "id": "quantum-explanation",
    "category": "reasoning",
    "description": "Test ability to explain complex concepts simply",
    "messages": [
      {
        "role": "user",
        "content": "Explain quantum computing in simple terms."
      }
    ]
  }
]

Evaluation System

The LLM-as-a-Judge evaluation system provides:

• Automated Scoring: AI-powered evaluation of response quality
• Statistical Analysis: Comprehensive metrics and distributions
• Model Rankings: Performance comparisons across all dimensions
• Interactive Reports: Web-based visualizations of results
• Detailed Critiques: Specific feedback for each response

Example Analysis Output

[Monitor] EVALUATION ANALYSIS RESULTS
══════════════════════════════════════════════════════════════

 OVERVIEW
• Total Evaluations: 150
• Models Tested: 3
• Prompts Tested: 5
• Success Rate: 94.7%

🏆 MODEL RANKINGS
1. openai:gpt-4o        - Avg Score: 8.7/10 (±0.8)
2. anthropic:claude-3-opus - Avg Score: 8.4/10 (±0.9)
3. blablador:gpt-4o     - Avg Score: 7.9/10 (±1.1)

 DETAILED METRICS
• Response Quality: 8.3/10 average
• Relevance: 8.6/10 average
• Accuracy: 9.1/10 average
• Creativity: 7.8/10 average

Latest Benchmark Results

Recent benchmarking results from the automated weekly workflow testing BLABLADOR models:

Model Performance Overview

Overall Statistics:

• Total Evaluations: 8 across 4 different prompts
• Models Tested: 2 BLABLADOR models
• Overall Success Rate: 100.0%
• Average Rating: 8.0/10
• Average Latency: 7.45s

Key Findings

• Top Performer: GPT-OSS-120b with highest rating (8.5/10) and fastest response time (5.35s)
• Most Consistent: GPT-OSS-120b with lower rating variation (±2.38)
• Performance Gap: 1.0 point difference between best and worst performing models
• Model Availability: Both tested models are fully operational with 100% success rates

Evaluation Details

• Prompt Categories: Reasoning, coding, and creative writing tasks
• Temperature Testing: Multiple temperature settings (0.1, 0.7, 1.0) for response variation
• LLM-as-a-Judge: Automated evaluation with detailed critiques and statistical analysis
• Rating Distribution: GPT-OSS-120b received mostly 9-10 ratings, Ministral-3-14B showed more variation

Reports and Visualizations

• Interactive HTML Report - Comprehensive evaluation analysis with charts
• Markdown Summary - Detailed performance metrics
• Performance Chart - Visual model comparison
• Basic Report - Simple performance overview

Reports are automatically updated and include LLM-as-a-Judge evaluation with detailed statistical analysis and model rankings.

Quality & Maturity

HeLLMholtz maintains enterprise-grade quality standards through comprehensive automated validation:

Quality Metrics

Automated Quality Gates

The production pipeline enforces 12 concurrent quality checks:

Maturity Indicators

• Versioning: Semantic Versioning (SemVer)
• Release Cadence: Automated on tag push
• Changelog: Automatic generation from commit messages
• Deprecation Policy: 6-month deprecation notice
• Backwards Compatibility: Strictly maintained
• Documentation: Versioned and auto-generated
• Support: Active community and enterprise support

Development

Setup Development Environment

# Clone repository
git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz

# Install with development dependencies
poetry install --with dev

# Install pre-commit hooks
poetry run pre-commit install

Running Tests

# Run all tests
poetry run pytest

# Run with coverage
poetry run pytest --cov=hellmholtz --cov-report=html

# Run specific test categories
poetry run pytest -m "slow"        # Slow integration tests
poetry run pytest -m "network"     # Tests requiring network access
poetry run pytest -m "model"       # Tests using actual models

Code Quality

HeLLMholtz maintains high code quality standards through comprehensive automated checks:

Quality Batches

# Execute all quality checks
poetry run ruff check .
poetry run ruff format . --check
poetry run mypy src/
poetry run pytest --cov=hellmholtz --cov-report=term-missing --cov-fail-under=50

Additional Quality Checks

# Security vulnerability scanning
poetry run bandit -r src/ -ll
poetry run safety check

# Documentation validation
poetry run doc8 docs/
poetry run pydocstyle src/

# Import organization
poetry run isort --check .

# Dependency auditing
poetry run pip-audit

CI/CD Quality Gates

The GitHub Actions pipeline enforces eight quality gates before deployment:

All quality gates must pass before packages are built or published.

Production Readiness

• Continuous Integration: Automated testing on every commit
• Continuous Deployment: Automated publishing to PyPI
• Version Tagging: Semantic versioning with automated changelog
• Artifact Management: Signed and verified package distributions
• Rollback Support: Version rollback capability
• Monitoring: Production deployment monitoring and alerting

Development

Setup Development Environment

# Clone repository
git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz

# Install with development dependencies
poetry install --with dev

# Install pre-commit hooks
poetry run pre-commit install

Running Tests

# Run all tests
poetry run pytest

# Run with coverage
poetry run pytest --cov=hellmholtz --cov-report=html

# Run specific test categories
poetry run pytest -m "slow"        # Slow integration tests
poetry run pytest -m "network"     # Tests requiring network access
poetry run pytest -m "model"       # Tests using actual models

Building Documentation

# Generate API documentation
poetry run sphinx-build docs/ docs/_build/

# Serve documentation locally
poetry run sphinx-serve docs/_build/

Contributing

We welcome contributions! Please see our Contributing Guide for details:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make your changes and add tests
4. Run the full test suite with quality checks
5. Commit your changes following conventional commits
6. Push to the branch: git push origin feature/amazing-feature
7. Open a Pull Request

All contributions must pass the CI/CD quality gates before merging.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Built on top of aisuite for unified LLM access
• LLM evaluation powered by EleutherAI's LM Evaluation Harness
• Proxy functionality via LiteLLM
• Special thanks to the Helmholtz Association for Blablador model access

Support

• Documentation: https://hellmholtz.readthedocs.io/
• Issue Tracker: https://github.com/JonasHeinickeBio/HeLLMholtz/issues
• Discussions: https://github.com/JonasHeinickeBio/HeLLMholtz/discussions
• Commercial Support: Available for enterprise deployments

Made with love for the scientific computing community

Made with love for the scientific computing community