giraffe-llm 0.1.0a1

Acceptance checks for existing LLM endpoints

Run repeatable tests against an existing OpenAI-compatible text endpoint. Inspect request evidence and compare results with a saved baseline.

Quick start · Inference engines · Baselines · Tests · Docs · Discord

Quick start

Start your inference server separately and have a model ready to use.

1. Install

The Python package is named giraffe-llm; the command is giraffe. Install from source until the first PyPI release:

git clone https://github.com/0dust/giraffe.git && cd giraffe
python3 -m venv .venv
source .venv/bin/activate
pip install .

2. Start your run

This example uses Ollama with llama3.2:3b already installed. Replace the URL and model for your server.

giraffe run --url http://127.0.0.1:11434/v1 \
  --model llama3.2:3b --checks access,serving,correctness \
  --concurrency 1 --html

This checks endpoint access, response validity and known-answer correctness, one request at a time.

Outputs: results in the terminal, plus JSON and HTML reports in runs/<timestamp>/. The command exits when finished.

Prefer the browser UI?

Run this in your terminal:

giraffe ui

Open localhost:8765, enter your endpoint and model, choose checks, then Launch. Keep the terminal open; Ctrl+C stops the UI. Use --port to choose a different port.

Inference engines

Connect through the engine’s OpenAI-compatible chat API. Discovery, metrics and server-side evidence vary by engine.

The YAML files below are included in the cloned repository’s examples/ directory. Each file defines the endpoint, model, checks, load settings and acceptance limits for a run.

Open the file for your engine and edit its URL, model, checks and limits. From the repository root, pass that file to --config. For example, after editing examples/vllm.yaml:

giraffe run --config examples/vllm.yaml \
  --output runs/before --html

Giraffe reads the run settings from YAML; --output chooses where to save the results, and --html adds an HTML report.

Save a baseline and compare

Save the completed runs/before run as a reference. After a deployment change, run the same configuration with --baseline:

giraffe baseline save runs/before baselines/before.json

# After changing your deployment:
giraffe run --config examples/vllm.yaml \
  --baseline baselines/before.json --output runs/after --html

The new report shows acceptance results and changes against the saved baseline. Keep the workload comparable; incompatible measurements stay inconclusive.

In the UI, open a completed run and choose Save as reference baseline, then Compare a new run with this baseline. Baseline guide →

Tests

15 built-in checks. Select the ones that matter to your endpoint.

Results are pass, fail, inconclusive, skipped or blocked. Missing delivery or measurement evidence stays inconclusive. Inspect the request evidence before attributing a failure to your inference server.

Documentation

Documentation index — detailed guides in the same repository:

Running & baselines · Workloads & measurements · Offline & Docker · Resource limits

Community

Join the Giraffe Discord to get help with runs and reports, share feedback, and discuss ideas. Start a post in #help-and-feedback for a question, bug, or suggestion, and follow #updates for releases and improvements.

Report an issue with your Giraffe version, engine/model and reproduction steps. Remove API keys and private prompts or answers before sharing reports.

Contributing · Apache 2.0 license