dpbench 0.2.0

DPBench: A Benchmark for LLM Multi-Agent Coordination

DPBench

A benchmark for evaluating coordination in multi-agent LLM systems under simultaneous resource contention.

Overview

DPBench adapts the Dining Philosophers problem into a controlled testbed where the action protocol, the communication structure, and the group size each vary independently. Each episode reports four metrics (deadlock rate, throughput, fairness, message-action consistency) with Wilson and t-based 95% confidence intervals. On a single model, the benchmark captures what changes when the protocol changes.

Installation

pip install dpbench

Optional provider extras for the example experiments:

pip install "dpbench[openai]"
pip install "dpbench[anthropic]"
pip install "dpbench[google]"
pip install "dpbench[xai]"

Quickstart

from dpbench import Benchmark

def my_model(system_prompt: str, user_prompt: str) -> str:
    """Your LLM call here. Returns the agent's response as a string."""
    ...

results = Benchmark.run(
    model_fn=my_model,
    system_prompt="...",
    decision_prompt="...",
    philosophers=5,
    episodes=30,
    mode="simultaneous",
    communication=False,
)

print(f"Deadlock rate: {results['deadlock_rate']:.1%}")
print(f"Throughput:    {results['avg_throughput']:.3f}")
print(f"Fairness:      {results['avg_fairness']:.3f}")

Prompt templates used in the paper are in experiments/prompts/. Full parameter documentation is in the Benchmark.run docstring.

Reproducing the paper experiments

git clone https://github.com/najmulhasan-code/dpbench.git
cd dpbench
pip install -e .

# Provider API keys for the LLMs you want to evaluate
cp .env.example .env

python experiments/scripts/run.py
python experiments/scripts/aggregate.py
python experiments/scripts/generate_figures.py

Experiments are configured by experiments/configs/conditions.yaml and experiments/configs/models.yaml.

Citation

A BibTeX entry will be added here when the accompanying paper is published.

License

Licensed under the MIT License.