sayf-eval

A lightweight, model-agnostic framework for evaluating LLMs on cybersecurity benchmarks.

sayf (Arabic: sword) -eval evaluates any LLM — hosted API or local checkpoint — through one common interface. It rests on two layers kept separate:

• Transport — LiteLLM: one Model adapter for every provider (OpenAI, Anthropic, Azure, …) and any OpenAI-compatible local server (vLLM via a base_url). No per-provider glue.
• Structure — lighteval-shaped: a Task / Model / Scorer boundary with a two-level metric split (sample-level extract+verdict, corpus-level aggregation).

The judge is not special — it is another Model, so the model-under-test and the judge can each be any provider with no code change.

Architecture

Task (prompt + params + dataset)
   └─> Model.generate(messages, params)        # LiteLLM under the hood
          └─> Scorer
                judge: Model                    # same type as the model
                extract + verdict (sample level)
                aggregate (corpus level)        # accuracy, VSP MAD, set F1

Install

pip install sayf-eval
# or, from source:
pip install -e ".[dev]"

Quick start

# Credentials: LiteLLM reads provider keys from the environment.
export OPENAI_API_KEY=...   ANTHROPIC_API_KEY=...

# End-to-end: inference + judge across a few tasks
sayf-eval run \
  --tasks mcq seceval vsp taa \
  --model openai/gpt-4o \
  --judge anthropic/claude-sonnet-4-20250514 \
  --output-dir outputs/gpt4o \
  --max-samples 5

# Or split the steps
sayf-eval run-inference --tasks mcq --model openai/gpt-4o --output-dir outputs/gpt4o
sayf-eval run-judge     --tasks mcq --judge openai/gpt-4o --output-dir outputs/gpt4o

Outputs per task: <task>_responses.jsonl, <task>_detailed.jsonl, and a combined summary.json.

Local models (vLLM)

A local model is just another endpoint: serve it OpenAI-compatibly and point sayf-eval at its base_url.

# Serve (tuning flags that used to live in scripts now live at serve time):
vllm serve Qwen/Qwen3-8B --port 8000 --enforce-eager

# Evaluate through the same interface (note the hosted_vllm/ prefix + base-url):
sayf-eval run \
  --tasks mcq vsp \
  --model hosted_vllm/Qwen/Qwen3-8B --base-url http://localhost:8000/v1 --api-key EMPTY \
  --judge anthropic/claude-sonnet-4-20250514 \
  --output-dir outputs/qwen3-8b

For reasoning models, pass --answer-stop to apply a stop sequence to the answer after the <think> block is stripped, and --max-tokens to scale the budget.

Tasks

25 cybersecurity sub-tasks across 9 benchmark families (sayf-eval run --tasks …):

• CTI-Bench: mcq, rcm, vsp, ate, cti_taa
• AthenaBench: ckt, rms, taa, athena_ate, athena_rcm, athena_vsp
• SECURE: secure_maet, secure_cwet, secure_kcv
• RedSage: redsage_frameworks, redsage_generals, redsage_skills, redsage_cli, redsage_kali
• Other MCQ: seceval, cybermetric, secbench, mmlu-cs, cissp
• SEvenLLM: sevenllm (open-ended structured CTI extraction / analysis, judged semantically)

cissp needs a dataset path via SAYF_EVAL_CISSP_PATH (not a public dataset); all others load from HuggingFace / GitHub on first run.

Standardized pipeline choices

sayf-eval applies fixed, documented choices that remove measurement artifacts without changing task semantics: temperature 0 / top_p 1 / fixed seed; per-task token budgets; <think> stripped before judging with the stop sequence applied to the answer only; and denominator = all attempted items (unparseable/empty answers are incorrect; only judge-API failures are excluded — from both numerator and denominator).

Results & leaderboard

Every sayf-eval run writes a canonical results record to <output-dir>/results/<model>/results_<ts>.json. Because scores are pipeline-dependent, the record embeds the full pipeline configuration (decoding params, token-budget policy, <think> handling, denominator policy, judge model) next to the per-task metrics — so entries are comparable by construction, not bare numbers.

Optionally push to a HuggingFace dataset (pip install 'sayf-eval[hub]'):

sayf-eval run --tasks mcq vsp --model openai/gpt-4o --judge openai/gpt-4o \
  --output-dir outputs/gpt4o \
  --results-org my-org --push-scores      # private dataset by default

Security posture (this is a cybersecurity benchmark):

Nothing is pushed without an explicit flag. The scores artifact never contains prompt or answer text, so it is safe to make public; details stay private regardless of --public.

Community leaderboard (HF Community-Evals)

Level 2 emits the two artifacts HuggingFace aggregates into a rendered leaderboard (docs) — opt-in, and a deliberate disclosure decision for security tasks.

# 1. Benchmark spec: register sayf-eval as a HF benchmark dataset (one
#    sub-leaderboard per task). Writes eval.yaml; --push-to creates the dataset.
sayf-eval benchmark-spec --out eval.yaml            # all 25 tasks
sayf-eval benchmark-spec --push-to qcri/sayf-eval --public   # private without --public

# 2. Per-model results: turn a results record into .eval_results/*.yaml and
#    (optionally) open a community PR to the model repo so scores show on its card.
sayf-eval eval-results --results outputs/gpt4o/results/openai__gpt-4o/results_*.json \
  --benchmark-id qcri/sayf-eval --out .eval_results/sayf-eval.yaml \
  --submit-pr openai/gpt-4o

Each .eval_results entry carries the pipeline config in its notes, so the public leaderboard never shows a bare number. Two one-time HF steps are required to go live (both noted by the CLI): the sayf-eval evaluation_framework must be added to HF's enum, and the benchmark dataset allow-listed (registration is beta).

Development

make install     # pip install -e ".[dev]"
make style       # ruff format + ruff check --fix
make quality     # ruff format --check + ruff check  (CI gate)
make test        # pytest

See CONTRIBUTING.md to get involved.

License

MIT.