llmdep 0.1.1

Distributed LLM Evaluation Benchmark Server

llmdep - Distributed LLM Evaluation Benchmark Server

A distributed benchmark server for evaluating Large Language Models (LLMs) across multiple benchmarks.

Features

• Distributed Evaluation: Support concurrent worker processes
• Multiple Benchmarks: ARC, ETHICS, MMLU-Pro, BiPaR, and more
• Rate Limiting: Token bucket based rate limiting for API calls
• Batch Processing: Process multiple model-benchmark pairs in batch
• Resume Support: Continue interrupted tasks with --run_id

Installation

pip install llmdep

Or install from source:

pip install -e .

Quick Start

Single Task

# Generate answers and evaluate
llmdep gen_eval -b ARC -m qwen-plus -w 10

Batch Processing

Create a task configuration file tasks.json:

[
    {"model": "qwen-plus", "benchmark": "ARC_parquet"},
    {"model": "deepseek", "benchmark": "ethics"}
]

Then run:

llmdep batch --config tasks.json -w 10

Resume Interrupted Task

llmdep gen_eval --run_id 20260210-164048-322281

Commands

gen_eval

Generate model answers and optionally evaluate.

llmdep gen_eval [options]

Options:
  -b, --benchmark     Benchmark name (e.g., ARC, ethics)
  -m, --model         Model name (e.g., qwen-plus, deepseek)
  -w, --worker_nums   Number of concurrent workers (default: 6)
  --batch_size        Batch size (default: 1)
  --rate_limit        Token bucket fill rate (default: 5.0)
  --bucket_capacity   Token bucket capacity (default: 20.0)
  --limited_test      Quick test with first 50 questions
  --evaluate         Run evaluation after generation
  --run_id          Resume from run_id

evaluate

Run evaluation on existing results.

llmdep evaluate -b ARC -r 20260210-164048-322281_qwen-plus

list

List available benchmarks or models.

llmdep list benchmark
llmdep list model

batch

Batch process tasks from config file.

llmdep batch --config tasks.json

Project Structure

llmdep/
├── llm/                  # LLM model implementations
│   ├── qwen_plus/
│   ├── deepseek/
│   └── ...
├── mep_client/           # Client for generating answers
│   ├── run/
│   └── API/
├── method_server/      # Benchmark implementations
│   ├── ARC_parquet/
│   ├── ethics/
│   ├── BiPaR/
│   └── ...
└── llmdep_cli.py       # CLI entry point

Adding New Models

1. Create folder under llm/<your_model/
2. Implement model.py inheriting from BaseModel
3. Create model_card.json
4. Register in llm/config.py

Adding New Benchmarks

1. Create folder under method_server/<benchmark_name>/
2. Implement prepare_ques.py - Prepare questions and golden answers
3. Implement eval_script.py - Run evaluation
4. Create dataset_card.json

License

MIT License