dqnEval

Deterministic, fully scriptable evaluation for LLMs. No LLM-based judging.

Features

• 250 question suite across instruction fidelity, programming, reasoning, context awareness, common sense, language & grammar, and tool calling
• OpenAI-compatible Chat Completions API support (local or hosted)
• JSON and Markdown reports
• Reproducible, code-driven scoring
• Suite validation and category descriptions

Quickstart

python -m venv .venv
source .venv/bin/activate
pip install -e .

dqneval list
dqneval describe
dqneval validate

dqneval run \
  --model gpt-4o-mini \
  --base-url https://api.openai.com/v1 \
  --api-key $OPENAI_API_KEY \
  --out-dir results

CLI

dqneval list

dqneval run --model MODEL --base-url URL --api-key KEY
  [--categories instruction_fidelity,programming]
  [--max-questions 100]
  [--out-dir results]
  [--format json,md]
  [--shuffle]
  [--seed 123]
  [--verbose]
  [--temperature 0]
  [--max-tokens 512]
  [--max-retries 2]
  [--request-delay 0.5]
  [--question-retries 2]
  [--retry-backoff 5]
  [--retry-max-delay 60]
  [--continue-on-error]
  [--api-key-header Authorization]
  [--api-key-prefix Bearer]

Environment defaults:
- OPENAI_BASE_URL
- OPENAI_API_KEY

Notes

• Prompts enforce JSON-only responses for deterministic parsing.
• Coding tasks run in a time-limited subprocess with tests.