Artemis LLM Benchmark

A Python CLI for correctness validation and performance benchmarking of LLM serving endpoints. Works with any OpenAI-compatible server — vLLM, Ollama, llama.cpp, and more.

Install

pip install artemisllmbench                    # core
pip install "artemisllmbench[dashboard]"       # + Streamlit dashboard
pip install "artemisllmbench[full]"            # + dashboard + semantic similarity

[full] adds sentence-transformers for semantic similarity checks (Layer 3 validity).

What It Does

Artemis LLM Benchmark answers two questions after you optimize an LLM endpoint:

• Did the optimization preserve correctness? — multi-layer validity checks on every response
• What is the publishable performance number? — reproducible latency, throughput, and goodput metrics

Quick Start

Validate a single endpoint — correctness + performance in one command:

artemisllmbench validate \
  --endpoint http://localhost:9000 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --hardware a100

Compare stock vs optimized — sequential benchmarking with full GPU resources for each:

artemisllmbench compare \
  --endpoint-a http://localhost:9000 \
  --endpoint-b http://localhost:9001 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --hardware a100

Split-session compare — when the stock endpoint is already torn down:

# Session 1: save the stock baseline
artemisllmbench baseline --endpoint http://localhost:9000 --model <model> --hardware a100

# Session 2: run the optimized candidate
artemisllmbench candidate --endpoint http://localhost:9000 --model <model> --hardware a100

The dashboard launches automatically after each run. Open it at http://<your-ip>:8501.

Key Features

• Multi-layer validity — sanity, structural, semantic (embedding similarity ≥ 0.92), and exact-match checks catch regressions that latency numbers alone miss
• Reproducible metrics — TTFT, P95/P99 latency, ITL (inter-token latency), throughput, CV, drift, and spike detection
• SLO / goodput tracking — set --slo-ttft and --slo-latency thresholds; get the % of requests that met them
• Streamlit dashboard — live progress, side-by-side results, analytics charts, and a live response comparison panel
• Fast mode — --fast cuts runtime by ~75% for quick iteration checks
• Cross-machine support — endpoints can be on different hosts or different hardware

Common Flags

Validity Layers

Pre-flight Conformance Check

artemisllmbench check-conformance --endpoint http://localhost:9000

Verifies your endpoint speaks the required OpenAI-compatible SSE format before a full benchmark run.

Full documentation and source: artemisllmbench --help or artemisllmbench <command> --help