HeLLMholtz LLM Suite

A comprehensive Python package for unified LLM access, benchmarking, evaluation, and reporting. Built on top of aisuite with specialized support for Helmholtz Blablador models.

Features

• Unified Client: Single interface for OpenAI, Google, Anthropic, Ollama, and Helmholtz Blablador models
• Centralized Configuration: Environment-based configuration for all your projects
• Advanced Benchmarking: Compare model performance across temperatures, replications, and prompt categories
• LLM-as-a-Judge Evaluation: Automated evaluation with comprehensive statistical analysis
• Interactive Reports: HTML reports with Chart.js visualizations and Markdown summaries
• Flexible Prompt System: Support for both simple text files and structured JSON prompt collections
• Model Monitoring: Track Blablador model availability and configuration consistency
• PII Anonymization: Document anonymization with the ShinrAI PII 1.3 model, including leak validation, cross-chunk consistency, and prompt-variant benchmarking
• LM Evaluation Harness: Integration with EleutherAI's comprehensive evaluation suite
• LiteLLM Proxy: Built-in proxy server for model routing and load balancing
• Throughput Testing: Performance benchmarking for high-throughput scenarios
• Model Discovery: Dynamic model listing and availability checking (19+ BLABLADOR models currently available)

System-One (Laya) CLI

The new hellm systemone command lets you route a state with custom questions and run a benchmark across clinical scenarios.

Usage examples

# Route a single state with a question
hellm systemone route "Patient presents with fatigue." -q "risk:Assess risk" -c "risk=low:Low;high:High"

# Run the benchmark (default 4 scenarios, 3 replications each)
hellm systemone benchmark --replications 3 --output-dir reports/systemone

Installation

Basic Installation

pip install hellmholtz

Development Installation

For development with all optional dependencies:

git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz
pip install -e ".[eval,proxy]"

Poetry Installation

poetry install --with eval,proxy

Configuration

HeLLMholtz supports both project-level and user-level configuration.

Project-Level Configuration (.env)

1. Copy the example environment file:

cp .env.example .env

2. Configure your API keys in .env:

# OpenAI
OPENAI_API_KEY=your_openai_key

# Anthropic
ANTHROPIC_API_KEY=your_anthropic_key

# Google
GOOGLE_API_KEY=your_google_key

# Helmholtz Blablador
BLABLADOR_API_KEY=your_blablador_key
BLABLADOR_API_BASE=https://your-blablador-instance.com

# Optional: Default models
AISUITE_DEFAULT_MODELS='{"openai": "gpt-4o", "anthropic": "claude-3-haiku"}'

User-Level Configuration (Recommended for pipx users)

For persistent configuration across all projects, use the hellm-setup tool:

# Configure once (works globally)
hellm-setup --api-key "your-blablador-key" \
            --base-url "https://api.blablador.ai/v1" \
            --default-models "meta-llama/Meta-Llama-3.1-8B-Instruct,gpt-4"

Configuration Location: ~/.config/hellmholtz/.env

This is automatically loaded for all hellm commands, regardless of your current directory.

Configuration Precedence: Project .env overrides user config (useful for per-project overrides).

See Configuration Guide for more details.

Usage

Python API

Basic Chat Interface

from hellmholtz.client import chat

# Simple chat
response = chat("openai:gpt-4o", "Hello, how are you?")
print(response)

# With conversation history
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."},
]
response = chat("anthropic:claude-3-sonnet", messages)

Benchmarking

from hellmholtz.benchmark import run_benchmarks
from hellmholtz.core.prompts import load_prompts

# Load prompts from JSON file
prompts = load_prompts("prompts.json", category="reasoning")

# Run benchmarks
results = run_benchmarks(
    models=["openai:gpt-4o", "anthropic:claude-3-haiku", "blablador:gpt-4o"],
    prompts=prompts,
    temperatures=[0.1, 0.7, 1.0],
    replications=3,
)

# Analyze results
from hellmholtz.evaluation_analysis import EvaluationAnalyzer

analyzer = EvaluationAnalyzer()
analysis = analyzer.analyze_evaluation_results("results/benchmark_latest.json")
analyzer.print_analysis_summary(analysis)

PII Anonymization

Anonymize documents with the ShinrAI PII 1.3 model (blablador:alias-anonymizer by default). Results include the anonymized text, the detected PII entities, and a leak validation report.

from hellmholtz.anonymizer import anonymize_file, anonymize_text

# Anonymize an in-memory text
result = anonymize_text("Patient John Doe was admitted on 12.03.2025.")
print(result.text)  # anonymized text
print(result.entities)  # PII entities that were replaced
print(result.validation.summary)  # leak validation summary

# Anonymize a file; writes <stem>.anonymized<suffix> plus a JSON sidecar
# (entities, mapping, validation) next to the output
result = anonymize_file("notes/patient.md", write=True)

# The entity mapping is reversible for authorized re-identification
restored = result.restore()

Longer documents are automatically split into word-safe chunks and reassembled with a consistent replacement mapping across chunk boundaries. Prompt variants (default, framed, strict, multilingual, footer_off) can be selected via the prompt keyword.

Command Line Interface

HeLLMholtz provides a comprehensive CLI for all operations:

Chat Interface

# Simple chat
hellm chat --model openai:gpt-4o "Explain the theory of relativity"

# Interactive mode
hellm chat --model anthropic:claude-3-sonnet --interactive

# With system prompt
hellm chat --model blablador:gpt-4o --system "You are a coding assistant" "Write a Python function to calculate fibonacci numbers"

Benchmarking

# Basic benchmark
hellm bench --models openai:gpt-4o,anthropic:claude-3-haiku --prompts-file prompts.txt

# Advanced benchmark with evaluation
hellm bench \
  --models openai:gpt-4o,blablador:gpt-4o \
  --prompts-file prompts.json \
  --prompts-category reasoning \
  --temperatures 0.1,0.7,1.0 \
  --replications 3 \
  --evaluate-with openai:gpt-4o \
  --results-dir results/

# Throughput testing
hellm bench-throughput \
  --model openai:gpt-4o \
  --requests 100 \
  --concurrency 10 \
  --prompt "Write a short story about AI"

Evaluation and Analysis

# Analyze benchmark results
hellm analyze results/benchmark_latest.json --html-report analysis_report.html

# Generate reports
hellm report --results-file results/benchmark_latest.json --output report.md

Model Management

# List available Blablador models
hellm models

# Monitor model availability and test accessibility
hellm monitor --test-accessibility

# Check model configuration consistency
hellm monitor --check-config

PII Anonymization

Anonymize documents with the ShinrAI PII 1.3 model (default: blablador:alias-anonymizer, override with --model). Leak validation and fuzzy matching run by default; disable with --no-validate / --no-fuzzy.

# Anonymize a file and print the anonymized text to stdout
hellm anonymize notes/patient.md

# Write the anonymized text next to the input (with JSON sidecar)
hellm anonymize notes/patient.md --write

# Read from stdin and print the detected entity table
cat notes/patient.md | hellm anonymize --show-entities

# Machine-readable JSON output
hellm anonymize notes/patient.md --json > patient_anonymized.json

# Compare all prompt variants across documents, scored against ground truth
hellm anonymize-benchmark \
  tests/anonymizer/fixtures/sample_short_en.md \
  tests/anonymizer/fixtures/sample_german.md \
  --ground-truth tests/anonymizer/fixtures/ground_truth.json

# Benchmark specific variants with more iterations; reports land in reports/anonymizer/
hellm anonymize-benchmark docs/ -p framed,strict --iterations 3

Benchmark reports include per-variant latency (p50/p95), throughput, validation pass rate, entity recall, and leak rate, written as Markdown and JSON to reports/anonymizer/.

Weekly Automated Benchmarking

The repository includes a GitHub Actions workflow (.github/workflows/weekly-benchmark.yml) that runs every Sunday at 00:00 UTC (and on demand via Run workflow) to answer two questions continuously:

1. Which models are available? Every registered Blablador model is probed against the live /models endpoint and tested for chat accessibility; results are written to models_status.yaml (availability, latency, category, last-checked timestamp).
2. How performant is each model on each task? All currently available chat models are benchmarked across every prompt category (reasoning, coding, creative, knowledge). The generated report contains an overall per-model summary plus a model × task performance matrix (success rate and average latency per category), so you can see at a glance which model is best for which kind of task.

Non-chat endpoints (embedding / transcription models) are detected and excluded from the benchmark automatically. If a run is interrupted, the workflow falls back to the runner's incremental partial results so the weekly report still reflects the completed requests.

To enable automated benchmarking, set repository secrets:

• BLABLADOR_API_KEY: Your Blablador API key
• BLABLADOR_API_BASE: Blablador API base URL (optional)

Outputs are committed back to the repository under reports/:

File	Content
weekly_benchmark_comprehensive.md / .html	Availability status, per-model summary, per-task success-rate & latency matrices, findings
weekly_benchmark_report.md / .html	Response-level benchmark detail (latency, tokens, ratings)
weekly_benchmark_chart.png	Performance visualization
weekly_benchmark_results.json	Raw result records for the week (regenerate reports anytime)

The same reports can be produced locally from any benchmark run:

# Full benchmark across all chat models and prompt categories
hellm bench --models blablador:fast,blablador:large,... --all-prompts --replications 3

# Availability check (updates models_status.yaml)
hellm monitor --test-accessibility

# Comprehensive weekly-style report (availability + per-task matrix)
python scripts/generate_comprehensive_report.py results/benchmark_<latest>.json

# Standard report + chart
hellm report results/benchmark_<latest>.json --format html-detailed --output reports/report.html
hellm chart results/benchmark_<latest>.json --output reports/chart.png

A companion workflow (daily-model-check.yml, every day at 02:00 UTC) refreshes only models_status.yaml, so availability information between benchmark runs stays current.

Latest Benchmark Results (2026-09-30)

Full run: 4 Blablador models × 36 prompts × 3 replications (432 requests, all prompts succeeded). Quality scores are LLM-as-a-Judge ratings (1–10, judge: blablador:Muse Glimmer 30b, independent of the benchmarked models); see the driver section below for how to reproduce or extend them. Raw data: results/benchmark_2026-09-30T23-29-27.138617.partial.judged.jsonl.

Model	Judge mean	Judge median	Rated	p50 latency	p50 tokens/s
blablador:large	8.62	9.0	106/108	2.5 s	90
blablador:code	8.51	9.0	90/108	7.7 s	41
blablador:fast	8.09	9.0	106/108	2.8 s	145
blablador:huge	5.01	4.0	107/108	5.7 s	82

Mean judge rating per prompt category:

Category	fast	large	huge	code
reasoning	9.3	8.8	7.9	9.8
creative	8.7	8.7	5.1	8.3
coding	8.3	9.4	4.3	9.2
knowledge	6.1	7.6	2.8	7.3

⚠️ blablador:huge is a persistent outlier this run — strongest on reasoning, but weak on knowledge/coding: many responses are truncated or stop after an empty <think> planning trace with no actual answer. blablador:code has 18 empty responses (excluded from judging). Both warrant investigation in the next cycle.

LLM-as-a-Judge for Existing Result Files

hellm bench --evaluate-with only judges freshly produced runs. To retroactively score an existing results file (1–10 rating + critique per response), use the resumable driver:

python scripts/run_llm_judge.py results/benchmark_<latest>.jsonl \
    --model "blablador:Muse Glimmer 30b" --workers 4

# Smoke test on a handful of records first:
python scripts/run_llm_judge.py results/benchmark_<latest>.jsonl --limit 4

Behaviour:

• Writes <results>.judged.jsonl (override with --output); the input file is never modified.
• Resumable: re-running keeps existing judgements (matched by a content hash of each record) and only retries records still missing a rating — safe after transient 5xx or unparseable judge replies; checkpoints are written atomically every 10 completions.
• Identical records (same model, params, prompt and response text) are judged once and the verdict is shared across duplicates.
• Choose a judge model that is not one of the benchmarked models to avoid self-preference bias. Reasoning-style judges may hide the verdict in reasoning_content; the driver handles that and can be nudged with --judge-max-tokens.

The judged file can then be fed to the report generators above, which pick up the ratings.

Advanced Features

# Run LM Evaluation Harness
hellm lm-eval \
  --model openai:gpt-4o \
  --tasks hellaswag,winogrande \
  --limit 100

# Start LiteLLM proxy server
hellm proxy \
  --config litellm_config.yaml \
  --port 8000

Project Structure

hellmholtz/
├── cli.py                 # Command-line interface
├── client.py              # Unified LLM client
├── monitoring.py          # Model availability monitoring
├── evaluation_analysis.py # Statistical analysis and reporting
├── export.py              # Result export utilities
├── core/
│   ├── config.py          # Configuration management
│   └── prompts.py         # Prompt loading and validation
├── benchmark/
│   ├── runner.py          # Benchmark execution
│   ├── evaluator.py       # LLM-as-a-Judge evaluation
│   └── prompts.py         # Benchmark-specific prompts
├── providers/
│   ├── blablador_provider.py # Custom Blablador provider
│   ├── blablador_config.py   # Blablador model configuration
│   ├── blablador.py          # Blablador utilities
│   └── __init__.py
├── reporting/
│   ├── html.py            # HTML report generation
│   ├── markdown.py        # Markdown report generation
│   ├── stats.py           # Statistical calculations
│   ├── utils.py           # Reporting utilities
│   └── templates/         # HTML templates
└── integrations/
    ├── lm_eval.py         # LM Evaluation Harness integration
    └── litellm.py         # LiteLLM proxy integration

Prompt System

HeLLMholtz supports two prompt formats:

Simple Text Format (prompts.txt)

What is the capital of France?
Explain quantum computing in simple terms.
Write a Python function to reverse a string.

Structured JSON Format (prompts.json)

[
  {
    "id": "capital-france",
    "category": "knowledge",
    "description": "Test basic geographical knowledge",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France?"
      }
    ],
    "expected_output": "Paris"
  },
  {
    "id": "quantum-explanation",
    "category": "reasoning",
    "description": "Test ability to explain complex concepts simply",
    "messages": [
      {
        "role": "user",
        "content": "Explain quantum computing in simple terms."
      }
    ]
  }
]

Evaluation System

The LLM-as-a-Judge evaluation system provides:

• Automated Scoring: AI-powered evaluation of response quality
• Statistical Analysis: Comprehensive metrics and distributions
• Model Rankings: Performance comparisons across all dimensions
• Interactive Reports: Web-based visualizations of results
• Detailed Critiques: Specific feedback for each response

Example Analysis Output

[Monitor] EVALUATION ANALYSIS RESULTS
══════════════════════════════════════════════════════════════

 OVERVIEW
• Total Evaluations: 150
• Models Tested: 3
• Prompts Tested: 5
• Success Rate: 94.7%

🏆 MODEL RANKINGS
1. openai:gpt-4o        - Avg Score: 8.7/10 (±0.8)
2. anthropic:claude-3-opus - Avg Score: 8.4/10 (±0.9)
3. blablador:gpt-4o     - Avg Score: 7.9/10 (±1.1)

 DETAILED METRICS
• Response Quality: 8.3/10 average
• Relevance: 8.6/10 average
• Accuracy: 9.1/10 average
• Creativity: 7.8/10 average

Latest Benchmark Results

Recent benchmarking results from the automated weekly workflow testing BLABLADOR models:

Model Performance Overview

Model	Success Rate	Avg Latency	Avg Rating (1-10)	Rating Std Dev
GPT-OSS-120b	100.0%	5.35s	8.5	±2.38
Ministral-3-14B-Instruct-2512	100.0%	9.55s	7.5	±3.70

Overall Statistics:

• Total Evaluations: 8 across 4 different prompts
• Models Tested: 2 BLABLADOR models
• Overall Success Rate: 100.0%
• Average Rating: 8.0/10
• Average Latency: 7.45s

Key Findings

• Top Performer: GPT-OSS-120b with highest rating (8.5/10) and fastest response time (5.35s)
• Most Consistent: GPT-OSS-120b with lower rating variation (±2.38)
• Performance Gap: 1.0 point difference between best and worst performing models
• Model Availability: Both tested models are fully operational with 100% success rates

Evaluation Details

• Prompt Categories: Reasoning, coding, and creative writing tasks
• Temperature Testing: Multiple temperature settings (0.1, 0.7, 1.0) for response variation
• LLM-as-a-Judge: Automated evaluation with detailed critiques and statistical analysis
• Rating Distribution: GPT-OSS-120b received mostly 9-10 ratings, Ministral-3-14B showed more variation

Reports and Visualizations

• Interactive HTML Report - Comprehensive evaluation analysis with charts
• Markdown Summary - Detailed performance metrics
• Performance Chart - Visual model comparison
• Basic Report - Simple performance overview

Reports are automatically updated and include LLM-as-a-Judge evaluation with detailed statistical analysis and model rankings.

Quality & Maturity

HeLLMholtz maintains enterprise-grade quality standards through comprehensive automated validation:

Quality Metrics

Metric	Value	Description
Test Coverage	≥50%	Code coverage across all modules
Type Coverage	100%	Full type annotation coverage
Security Score	A+	Zero vulnerabilities detected
Documentation	100%	Complete API and user documentation
Code Style	ruff	Industry-standard formatting
CI Status	✅	All quality gates passing

Automated Quality Gates

The production pipeline enforces 12 concurrent quality checks:

Gate	Tool	Status
Syntax & Style	Ruff	✅ Required
Formatting	Ruff Format	✅ Required
Type Safety	Mypy	✅ Required
Unit Tests	pytest	✅ Required
Coverage	pytest-cov	✅ Required (≥50%)
Security Scan	Bandit	✅ Required
Vulnerability Audit	Safety	✅ Required
Doc Validation	doc8	✅ Required
Import Organization	isort	✅ Required
Dependency Audit	pip-audit	✅ Required
Code Complexity	pylint	✅ Recommended
Memory Safety	valgrind	✅ Optional

Maturity Indicators

• Versioning: Semantic Versioning (SemVer)
• Release Cadence: Automated on tag push
• Changelog: Automatic generation from commit messages
• Deprecation Policy: 6-month deprecation notice
• Backwards Compatibility: Strictly maintained
• Documentation: Versioned and auto-generated
• Support: Active community and enterprise support

Development

Setup Development Environment

# Clone repository
git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz

# Install with development dependencies
poetry install --with dev

# Install pre-commit hooks
poetry run pre-commit install

Running Tests

# Run all tests
poetry run pytest

# Run with coverage
poetry run pytest --cov=hellmholtz --cov-report=html

# Run specific test categories
poetry run pytest -m "slow"        # Slow integration tests
poetry run pytest -m "network"     # Tests requiring network access
poetry run pytest -m "model"       # Tests using actual models

Code Quality

HeLLMholtz maintains high code quality standards through comprehensive automated checks:

Quality Batches

# Execute all quality checks
poetry run ruff check .
poetry run ruff format . --check
poetry run mypy src/
poetry run pytest --cov=hellmholtz --cov-report=term-missing --cov-fail-under=50

Additional Quality Checks

# Security vulnerability scanning
poetry run bandit -r src/ -ll
poetry run safety check

# Documentation validation
poetry run doc8 docs/
poetry run pydocstyle src/

# Import organization
poetry run isort --check .

# Dependency auditing
poetry run pip-audit

CI/CD Quality Gates

The GitHub Actions pipeline enforces eight quality gates before deployment:

Gate	Tool	Purpose
Linting	Ruff	Detect code style issues and bugs
Formatting	Ruff	Ensure consistent code formatting
Type Checking	Mypy	Validate static type annotations
Unit Tests	pytest	Execute test suite with coverage
Security	Bandit/Safety	Scan for vulnerabilities
Documentation	doc8/pydocstyle	Validate docs quality
Imports	isort	Organize import statements
Dependencies	pip-audit	Audit package vulnerabilities

All quality gates must pass before packages are built or published.

Production Readiness

• Continuous Integration: Automated testing on every commit
• Continuous Deployment: Automated publishing to PyPI
• Version Tagging: Semantic versioning with automated changelog
• Artifact Management: Signed and verified package distributions
• Rollback Support: Version rollback capability
• Monitoring: Production deployment monitoring and alerting

Development

Setup Development Environment

# Clone repository
git clone https://github.com/JonasHeinickeBio/HeLLMholtz.git
cd HeLLMholtz

# Install with development dependencies
poetry install --with dev

# Install pre-commit hooks
poetry run pre-commit install

Running Tests

# Run all tests
poetry run pytest

# Run with coverage
poetry run pytest --cov=hellmholtz --cov-report=html

# Run specific test categories
poetry run pytest -m "slow"        # Slow integration tests
poetry run pytest -m "network"     # Tests requiring network access
poetry run pytest -m "model"       # Tests using actual models

Building Documentation

# Generate API documentation
poetry run sphinx-build docs/ docs/_build/

# Serve documentation locally
poetry run sphinx-serve docs/_build/

Contributing

We welcome contributions! Please see our Contributing Guide for details:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make your changes and add tests
4. Run the full test suite with quality checks
5. Commit your changes following conventional commits
6. Push to the branch: git push origin feature/amazing-feature
7. Open a Pull Request

All contributions must pass the CI/CD quality gates before merging.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Built on top of aisuite for unified LLM access
• LLM evaluation powered by EleutherAI's LM Evaluation Harness
• Proxy functionality via LiteLLM
• Special thanks to the Helmholtz Association for Blablador model access

Support

• Documentation: https://hellmholtz.readthedocs.io/
• Issue Tracker: https://github.com/JonasHeinickeBio/HeLLMholtz/issues
• Discussions: https://github.com/JonasHeinickeBio/HeLLMholtz/discussions
• Commercial Support: Available for enterprise deployments

---

Made with love for the scientific computing community

- Discussions: https://github.com/JonasHeinickeBio/HeLLMholtz/discussions

---

Made with love for the scientific computing community