DPBench

A benchmark for evaluating coordination in multi-agent LLM systems under simultaneous resource contention.

Overview

DPBench adapts the Dining Philosophers problem into a controlled testbed where the action protocol, the communication structure, and the group size each vary independently. Each episode reports four metrics (deadlock rate, throughput, fairness, message-action consistency) with Wilson and t-based 95% confidence intervals. On a single model, the benchmark captures what changes when the protocol changes.

Read the full paper on arXiv.

Installation

pip install dpbench

Optional provider extras for the example experiments:

pip install "dpbench[openai]"
pip install "dpbench[anthropic]"
pip install "dpbench[google]"
pip install "dpbench[xai]"

Quickstart

from dpbench import Benchmark

def my_model(system_prompt: str, user_prompt: str) -> str:
    """Your LLM call here. Returns the agent's response as a string."""
    ...

results = Benchmark.run(
    model_fn=my_model,
    system_prompt="...",
    decision_prompt="...",
    philosophers=5,
    episodes=30,
    mode="simultaneous",
    communication=False,
)

print(f"Deadlock rate: {results['deadlock_rate']:.1%}")
print(f"Throughput:    {results['avg_throughput']:.3f}")
print(f"Fairness:      {results['avg_fairness']:.3f}")

Prompt templates used in the paper are in experiments/prompts/. Full parameter documentation is in the Benchmark.run docstring.

Reproducing the paper experiments

git clone https://github.com/najmulhasan-code/dpbench.git
cd dpbench
pip install -e .

# Provider API keys for the LLMs you want to evaluate
cp .env.example .env

python experiments/scripts/run.py
python experiments/scripts/aggregate.py
python experiments/scripts/generate_figures.py

Experiments are configured by experiments/configs/conditions.yaml and experiments/configs/models.yaml.

Citation

If you use DPBench in your work, please cite:

@misc{hasan2026dpbenchstructuraldeterminantsmultiagent,
      title={DPBench: Structural Determinants of Multi-Agent LLM Coordination Under Simultaneous Resource Contention},
      author={Najmul Hasan and Prashanth BusiReddyGari},
      year={2026},
      eprint={2602.13255},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2602.13255},
}

License

Licensed under the MIT License.