llmdep - Distributed LLM Evaluation Benchmark Server

A distributed benchmark server for evaluating Large Language Models (LLMs) across multiple benchmarks.

Features

• Distributed Evaluation: Support concurrent worker processes
• Multiple Benchmarks: ARC, ETHICS, MMLU-Pro, BiPaR, and more
• Rate Limiting: Token bucket based rate limiting for API calls
• Batch Processing: Process multiple model-benchmark pairs in batch
• Resume Support: Continue interrupted tasks with --run_id

Installation

pip install llmdep

Or install from source:

pip install -e .

Quick Start

Configure Benchmark Workspace

Benchmarks should live outside the installed pip package. Configure a benchmark root once:

llmdep config set dataset_root /data/eval_datasets

The directory should contain benchmark folders directly:

/data/eval_datasets/
├── BiPaR/
│   ├── dataset_card.json
│   ├── methods/
│   ├── data/
│   └── cache/
└── gsm8k/
    ├── dataset_card.json
    ├── methods/
    ├── data/
    └── cache/

Check the active benchmark root:

llmdep list benchmark

Configure a model root once as well:

llmdep config set model_root /data/llm_models

The directory should contain only user model folders directly. BaseModel is not placed here; it remains inside the installed llmdep package.

/data/llm_models/
└── qwen-plus/
    ├── model.py
    └── model_card.json

Each model.py should expose a Model class with call_with_prompt(...). Import the base class from the installed package:

from llm.BaseModel import BaseModel

class Model(BaseModel):
    def call_with_prompt(self, input_data):
        ...

Single Task

# Generate answers
llmdep gen_eval -b ARC -m qwen-plus -w 10

# Generate answers and evaluate
llmdep gen_eval -b ARC -m qwen-plus -w 10 --evaluate

Batch Processing

Create a task configuration file tasks.json:

[
    {"model": "qwen-plus", "benchmark": "ARC_parquet"},
    {"model": "deepseek", "benchmark": "ethics"}
]

Then run:

llmdep batch --config tasks.json -w 10

Resume Interrupted Task

llmdep gen_eval --run_id 20260210-164048-322281

Commands

gen_eval

Generate model answers and optionally evaluate.

llmdep gen_eval [options]

Options:
  -b, --benchmark     Benchmark name (e.g., ARC, ethics)
  -m, --model         Model name (e.g., qwen-plus, deepseek)
  -w, --worker_nums   Number of concurrent workers (default: 2)
  --batch_size        Batch size (default: 1)
  --rate_limit        Token bucket fill rate (default: 5.0)
  --bucket_capacity   Token bucket capacity (default: 10.0)
  --limited_test      Quick test with first 10 questions, or pass N
  --evaluate         Run evaluation after generation
  --run_id          Resume from run_id

evaluate

Run evaluation on existing results.

llmdep evaluate -b ARC -r 20260210-164048-322281_qwen-plus

list

List available benchmarks or models.

llmdep list benchmark
llmdep list model

dashboard

Start the local web control panel.

llmdep dashboard
llmdep dashboard --host 127.0.0.1 --port 8765

batch

Batch process tasks from config file.

llmdep batch --config tasks.json

Project Structure

llmdep/
├── llm/                  # Packaged BaseModel compatibility module
├── mep_client/           # Client for generating answers
│   ├── run/
│   └── API/
├── method_server/      # Benchmark implementations
│   ├── ARC_parquet/
│   ├── ethics/
│   ├── BiPaR/
│   └── ...
└── llmdep_cli.py       # CLI entry point

Adding New Models

1. Configure model_root: llmdep config set model_root /data/llm_models
2. Create folder under /data/llm_models/<your_model>/
3. Implement model.py inheriting from BaseModel
4. Create model_card.json

Adding New Benchmarks

1. Configure dataset_root: llmdep config set dataset_root /data/eval_datasets
2. Create folder under /data/eval_datasets/<benchmark_name>/
3. Implement prepare_ques.py - Prepare questions and golden answers
4. Implement eval_script.py - Run evaluation
5. Create dataset_card.json

License

MIT License