Hermia

Structured behavioral eval for local LLMs. The model binary is not the unit of analysis — the inference stack is.

You selected a model by benchmark score. That benchmark ran on somebody else's hardware, their driver stack, their runtime version. Not yours.

A ROCm update can flip a security test from PASS to FAIL. Hermia catches it — because it runs on your stack, not a cloud proxy.

What It Does

Hermia runs structured behavioral evaluation against local Ollama models and scores results for correctness across security, reasoning, and tool-use dimensions. Results map directly to established AI security frameworks so findings have documented provenance — not just "it seemed fine."

Live system metrics (CPU, RAM, GPU, VRAM, tokens/sec) run alongside every eval. Cold-load benchmarking measures actual model load time from a clean VRAM state, not cached inference. Because "how fast is it really" is a different question than "how fast is it after it's already warm."

v0.2 scope: structural eval with deterministic orchestration (fixed sampling — temperature=0, seed=42 — and fixed message construction) against Ollama-compatible local endpoints, with multi-turn corpus cases for context-carry and boundary-persistence testing. Reproducibility of the model's output still depends on the backend; that is what Hermia measures. LLM-as-judge intent scoring lands in v0.3.

Fleet mode (--fleet FILE) runs headless multi-host eval from a YAML config — same test suite, multiple Ollama endpoints evaluated concurrently (default: up to 4 hosts in parallel). Compare CUDA vs. Metal on the same model. See where your inference stack diverges. Entries that share the same host are evaluated sequentially so a single GPU node is never asked to hold two models simultaneously (VRAM-safe). Control parallelism with --max-concurrency N. Per-test timeout is configurable via --test-timeout SECONDS or per-host test_timeout: in the fleet YAML. See the fleet-YAML format for the file schema.

Why Hermia Exists

Garak is built by NVIDIA — you know, the company currently valued at roughly the GDP of a medium-sized country. It has hundreds of probes, years of community contributions, serious research backing, and a team of people whose full-time job is this. You should use it.

Hermia is built in a consultancy lab. Different scale. Genuinely different problem.

Garak asks: is this model vulnerable to known attack patterns?

Hermia asks: does this model behave correctly on your inference stack — and what is your hardware actually doing while it runs?

• Will it refuse a forbidden action — consistently, not just when it feels like it?
• Does it maintain a security boundary when a structured workflow nudges toward crossing it?
• Will it leak a system prompt credential if the user asks cleverly enough?
• Does it correctly route a request that looks safe but isn't?

These aren't hypothetical. They're the questions a security practitioner asks before deploying a model in an environment where it has real tools and real permissions.

Garak scans for vulnerabilities. Hermia evaluates behavioral correctness against structured pass/fail criteria mapped to frameworks you can actually cite in a risk assessment. They do different things. Run both.

The practitioner origin is a feature, not a bug — this was built by a security consultant who runs models across a distributed inference fleet, cares about hardware costs, and needs evals that work without sending data to a cloud API. If that sounds like you, Hermia was built for your context.

Framework Coverage

Eval Dimensions

Requirements

• Python 3.11+
• Ollama running locally (ollama serve)
• At least one model pulled: ollama pull llama3.2 or any compatible model

In the default local setup, no cloud API keys are required and no data leaves your machine. (Point a fleet host at a remote or cloud endpoint via the openai-compat transport and prompts are sent to that endpoint — and a key may be required. See Run against a remote host.)

Hardware Support

*NVIDIA metrics tested on Linux eval client. Windows Ollama servers are supported as fleet targets (point a fleet YAML entry's host: at the Windows box); running Hermia itself on Windows is not yet supported.

Install

Recommended (via pipx):

pipx install hermia

Or via Homebrew (macOS):

brew install scottblydotcom/tap/hermia

Or with pip:

pip install hermia

Or from source:

git clone https://github.com/scottblydotcom/hermia
cd hermia
pip install -e .

Or via Docker (headless fleet mode):

mkdir -p results && chmod 777 results  # container writes as uid 1000, not your host user
docker run --rm --network host \
  -v $PWD/fleets:/workspace/fleets:ro \
  -v $PWD/results:/workspace/results \
  ghcr.io/scottblydotcom/hermia:latest \
  --fleet fleets/local.yaml

See Docker usage for macOS / Windows networking (host.docker.internal) and volume-mount details.

Quickstart

# Start Ollama if it isn't running
ollama serve

# Launch Hermia
hermia

Hermia opens a TUI. Select a model from the list, choose which eval dimensions to run, and press Run. Results appear live alongside system metrics. Each run writes results/eval_TIMESTAMP.jsonl and results/eval_TIMESTAMP.csv.

New here? docs/getting-started.md is the 5-minute zero-to-first-eval path.

See docs/usage.md for the full reference: result interpretation, --repeat N consistency scoring, fleet mode, regression detection, and Postgres export.

Roadmap

v0.2 — Fleet + TUI (a.k.a. Endpoint Bus; shipping): Headless fleet mode for multi-host eval from a YAML config; full-featured TUI for launch/configure/run/inspect; backend stack tagging by GPU arch, runtime version, and execution path (GPU vs spill). Configurable per-test timeout for thinking-mode models.

v0.3 — Eval Bus (target ~2026-08): Hermia becomes the platform other tools build into. Probe adapters for Garak, PyRIT, and HarmBench pull their results into Hermia's hardware-correlated, framework-mapped view alongside Hermia's own test cases. LLM-as-judge scoring; a Sink interface — a pluggable output destination (Prometheus, webhook, S3) that results can be written to.

See docs/roadmap.md for the full plan.

Project Status

v0.2.1 — stable and tested. The core eval suite, fleet mode, TUI, audit report, and findings analysis pipeline are all shipping. Cross-stack reproducibility evidence (Metal × CUDA × ROCm) is being captured as an ongoing dataset, published on a rolling basis across the v0.2.x series rather than as a single launch snapshot. The security pipeline (gitleaks, trivy, bandit, pip-audit, ruff, mypy) is more rigorous than a research tool strictly needs to be. That was intentional.

Available on PyPI: pipx install hermia

Name

Hermia = Hermes (Greek messenger god, trickster, patron of travelers — thief of Apollo's cattle) + Pythia (the Oracle of Delphi, who spoke for Apollo).

The tool steals answers from the Oracle and tells you which one to trust.

Documentation

• Getting Started — 5-minute zero-to-first-eval guide
• Usage Reference — full walkthrough: install, run, interpret results, fleet mode, regression detection, Postgres export
• Roadmap — v0.2 fleet + TUI, v0.3 eval bus, full backlog
• GUARDS Framework — six-dimension standard for LLM system-prompt guardrail construction (Goal/Unit/Actions/Response/Detect/Stop)

Security

Hermia only reads from Ollama — /api/tags, /api/generate, /api/ps, and /api/version. It never calls the model-upload / /api/create endpoints, so it does not itself exercise the code paths behind the model-upload CVEs (CVE-2026-7482, CVE-2026-5757). Your Ollama server can still be vulnerable — keep it patched and restricted per the checklist below.

Protect your Ollama instance:

• Run Ollama bound to 127.0.0.1 (the default) — never expose port 11434 publicly
• Keep Ollama upgraded; 0.17.1+ patches CVE-2026-7482 (CVSS 9.1, heap memory disclosure via crafted GGUF upload, nicknamed "Bleeding Llama")
• CVE-2026-5757 (same attack class, no upstream patch as of May 2026) — restrict /api/create access at the network or firewall layer
• Fleet deployments: use fleet-YAML auth.bearer.key_env blocks (see usage.md) or a Tailscale overlay to prevent unauthenticated access to remote Ollama endpoints

Hermia surfaces known Ollama version vulnerabilities at run time in the preflight log as SEC ⚠ warnings.

Contributing

Contributions welcome. Please read AGENTS.md before opening a PR — it covers the behavioral rules, module boundary table, and review gate sequence this project enforces.

See CONTRIBUTING.md for full details on how to get involved.

License

MIT — see LICENSE.