Skip to main content

PACE: Parity-Aware Compute Evaluation

More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs

Author: Seyed Mahmoud Sajjadi Mohammadabadi
Affiliation: Department of Computer Science and Engineering, University of Nevada, Reno
Correspondence: mahmoud.sajjadi@unr.edu
Repository: https://github.com/mahmoudsajjadi/pace
arXiv Preprint: arXiv:2611.17610 (HTML version)
Paper PDF: Preprint: manuscript/main.pdf | Visual Cheatsheet: docs/assets/pace_cheatsheet.pdf | Interactive Studio: dist/hf_space_pace_studio/

arXiv:2611.17610 License: Apache 2.0 Data: CC BY 4.0 Python 3.10+ Test Suite: 865 Passing Zenodo DOI Open in Colab

Reproducible on one consumer GPU. The controlled experiments (llama3.1:8b served by Ollama, about 4.9 GB) run on a single 12 GB card (developed on an RTX 4070): bash scripts/reproduce_12gb_gpu.sh runs the tests, the hidden-profile file check and a 170-call smoke run, and records its own wall time (the expected runtimes in the script header are derived from a measured batch rate, not from a timed run of the script).

Official replication repository, audited contrast ledger (9 primary studies, 516 unique multi-agent configurations from 531 raw extracted rows, 834 contrast records), pace CLI and evaluation toolkit (legacy aliases: cme, mae-eval), and zero-dependency browser workbench (PACE Studio) for Parity-Aware Compute Evaluation.

Licences. Code (code/, analysis/, scripts/, tests/): Apache-2.0 (LICENSE). Dataset files (extracted contrast tables and the hidden-profile task, staged in zenodo_datasets.json): CC-BY-4.0.

PACE 60-second explainer


Abstract & Canonical Findings

Multi-agent language model evaluations often compare communicating systems against a single greedy pass, confounding inter-agent coordination with independent sampling and expanded test-time compute. We introduce Parity-Aware Compute Evaluation (PACE), which classifies single-agent baselines on a four-category ladder (Category A: single greedy pass $k=1$; Category B: noncommunicating sampled baseline; Category C: resource-matched or shared-ceiling baseline; Category D: compute-cost Pareto frontier), audits a resource vector with six dimensions

$$\mathbf{R} = \langle R_{\text{reason}}, R_{\text{token}}, R_{\text{cost}}, R_{\text{lat}}, R_{\text{sample}}, R_{\text{sys}} \rangle,$$

enforces directional resource feasibility, and measures the frontier gap relative to single-agent self-consistency at parity in tokens.

Extracted Corpus & Empirical Synthesis

  • Canonical Denominators: From 12 studies considered across two stages (5 initial studies and 7 screened candidates: 4 included, 3 excluded), the included corpus comprises 9 primary studies, 516 unique multi-agent configurations (531 raw extracted rows minus 15 baseline-only rows), and 834 baseline-specific contrast records (358 Category A, 360 Category B, 114 Category C, and 2 Category D Pareto contrasts; 474 combined Category B/C contrasts across 8 studies; 156 shared-ceiling contrasts; 318 paired Category A/B contrasts across 72 unique baseline cells).
  • Single-Pass vs. Compute-Controlled Categories: Mean accuracy differences are -0.032 against Category A (358 contrasts; -0.005 omitting Zhang et al., who supply 180 configurations), -0.051 against Category B (360 contrasts), and -0.012 against Category C (114 contrasts). Across combined Category B/C (474 contrasts, 8 studies), the unweighted contrast mean is -0.042, with exploratory three-level REML log odds $\hat\beta = \mathbf{-0.195}$ (CR2 95% CI [-0.323, -0.067]; 95% study-mean prediction interval [-0.515, +0.125]).
  • Consistency Across Studies: Multi-agent systems fail to lead in 74% (352 of 474) of Category B/C contrasts, feasible baselines match or beat them in 36% (171 of 474 directionally feasible; 56 below -1.96 SE; 69% or 356 of 516 unique configurations across Categories A/B/C), and 8 of 9 study means are nonpositive (sole positive exception: Wunderlich et al. at +0.020 at stated configurations on MMLU-Pro, compute not matched).
  • Sampling Gap ($\Delta_{\text{gap}}$): Across 5 dual-baseline studies (318 pairs spanning 72 unique baseline cells, 64 of 72 negative), switching from greedy to sampled baselines lowers accuracy differences by -0.038 weighted equally by study (-0.028 across pairs; -0.036 across the 72 unique cells; cell-level REML $\hat\beta = \mathbf{-0.178}$, CR2 95% CI [-0.347, -0.009]).
  • Controlled Open-Weight Experiment (llama3.1:8b via Ollama):
    • Shared-evidence MMLU-Pro (n=250): Vanilla debate (−0.015 frontier gap, 0.567 vs. 0.581 frontier) and Mixture-of-Agents (−0.072 frontier gap) do not clear the equal-token self-consistency frontier under Holm correction.
    • Complementary Hidden Profile split-view control (n=200): Split-view debate exceeds the split-view independent-voting frontier (+0.073 frontier gap, 0.182 vs. 0.109, Holm-adjusted p = 0.007), while trailing content-free placebo prompts (0.248) and four-choice chance (0.250).

Repository Structure

├── .github/                   # CI workflows & study-contrast issue template (add_study_contrast.yml)
├── analysis/                  # Synthesis, REML pooling, plotting, and LaTeX build gates
│   ├── assemble_paper.py      # Assembles & compiles manuscript/main.pdf
│   ├── run_baseline_tiers.py  # Descriptive corpus synthesis, dual-baseline pairing & numbers.tex
│   └── run_k9_reml.py         # Three-level REML + CR2 inference (including 72-cell Model D)
├── code/
│   ├── pace/                  # Standalone PACE CLI, importers (incl. Inspect AI), & browser Studio
│   ├── cme/                   # Legacy CLI/API aliases preserved for backwards compatibility
│   └── mae_eval/              # Legacy resource tracker/auditor package aliases
├── dist/
│   ├── assets/                # Explainer GIF & cheatsheet (PNG/PDF)
│   ├── github_release_staging/# Offline-verified GitHub release staging tree
│   └── hf_space_pace_studio/  # Hugging Face Space workbench + corpus_contrasts.json (832 rows)
├── docs/                      # Project landing page (index.html), community playbook & visual assets
├── evidence/                  # Verified extracted CSVs, screening ledger & audit reports
│   ├── contrasts_anchor5_core.csv
│   ├── contrasts_dual_baseline.csv
│   ├── extension_contrasts.csv
│   └── verification_worksheet.csv
├── figures/                   # Publication vector PDF figures and PNG previews
├── manuscript/                # Modular LaTeX source of truth and generated macro files
├── notebooks/                 # Interactive Colab/Jupyter log-auditing notebook (audit_your_log.ipynb)
├── outputs/                   # Baseline contrast tables, REML JSON & experimental logs
├── paper/                     # Synchronized manuscript build targets and figure/table mirrors
├── pyproject.toml             # PEP 621 package metadata (`pace-eval`) & CLI entry points
├── review/                    # Programmatic concordance ledgers & claim-verification tables
├── scripts/                   # Reproducibility runner (12 GB GPU), corpus builders & exporters
├── supplement/                # Moderator synthesis, empirical trace harvest & author templates
└── tests/                     # Automated unit, regression, and concordance test suite

Quickstart: the pace CLI & PACE Studio

Install pace-eval from the repository root (pyproject.toml) or from code/:

pip install -e .               # installs `pace` (plus legacy `cme` and `mae-eval` console aliases)

Core auditing, reporting, log import (including Inspect AI via --format inspect), and local GUI commands:

pace studio                    # launches the zero-dependency browser UI (PACE Studio)
pace classify --config sas.json
pace audit run_trace.jsonl
pace import-logs --format inspect --input eval_log.json --output trace.jsonl && pace audit trace.jsonl
pace report --pool pool.jsonl --logs run.jsonl --system debate --export-latex card.tex
pace dynamics debate_log.jsonl --null null.jsonl --json
pace card run_trace.jsonl --latex --output pace-card.tex

Constructive planning, simulation, meta-analysis, frontier, and rebuttal commands:

pace pool-size --agents 5 --rounds 3 --prompt-tokens 100 --msg-tokens 20 --json
pace simulate --mode debate --items 100 --agents 5 --rounds 3 --out debate.jsonl
pace meta --contrasts outputs/baseline_tier_contrasts.csv --scale log_or --json
pace power --p-base .55 --delta .10 --n-items 40 --json
pace benchmark-gen --route clue_grid --out clues.jsonl
pace aggregate-open --input candidates.jsonl --metric token_jaccard --out centroids.jsonl
pace plot --kind phase --out phase.svg
pace hidden-profile --n 100 --out hp.jsonl
pace frontier --samples pool.jsonl --logs run.jsonl --out frontier.pdf
pace rebuttal --contrasts data/baseline_tier_contrasts.csv --export-markdown rebuttal.md

End-to-end synthetic walkthrough, interactive Colab notebook, GitHub Action CI workflow template, and community study-submission playbook live in:

Legacy imports (from cme ... and from mae_eval ...) remain supported alongside pace.


Reporting Standard: Evaluation Card & Progressive Levels

Every completed comparison should report:

  1. The seven-field PACE Evaluation Card: Baseline rung (A–D), resource vector $\mathbf{R}$, token accounting (prompt, completion, cached vs. uncached tokens), MAS/SAS resource ratios, frontier gap with 95% paired bootstrap CI, evidence/route structure (shared vs. split, verifier availability, model heterogeneity), and per-item resource distribution and stopping rules.
  2. Three progressive reporting levels:
    • PACE-Minimum: Category B/C baseline (e.g., nested initial-round self-consistency voting), full token/call counts, and paired bootstrap accuracy intervals.
    • PACE-Recommended: Equal-token self-consistency frontier curve, directional feasibility across all six dimensions of $\mathbf{R}$, and placebo control or split-view checks.
    • PACE-Extended: Per-item budget matching ($k_i$), cache-weighted billing accounting, adaptive stopping distributions, hybrid call-graph edge counts, and peer-influence transition dynamics ($\tau_{i\to c}$ vs. $\tau_{c\to i}$).

Verification & Manuscript Compilation

Run the full automated verification suite (865 passed, 8 skipped):

pytest -q tests/

Assemble and verify all PDF targets (manuscript/main.pdf, paper/main.pdf) with Build Gates 1 and 2:

python -I analysis/assemble_paper.py

Citation

@article{sajjadi2026more,
  title   = {More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs},
  author  = {Sajjadi Mohammadabadi, Seyed Mahmoud},
  journal = {arXiv preprint arXiv:2611.17610},
  year    = {2026},
  eprint  = {2611.17610},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  doi     = {10.48550/arXiv.2611.17610},
  url     = {https://arxiv.org/abs/2611.17610},
  note    = {Replication repository at https://github.com/mahmoudsajjadi/pace}
}

Metadata

Release files for pace-eval 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pace-eval 0.2.2
File Size Uploaded Provenance Actions
pace_eval-0.2.2.tar.gz 245.5 kB Without provenance Details for pace_eval-0.2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pace-eval 0.2.2
File Interpreter ABI Platform Actions
pace_eval-0.2.2-py3-none-any.whl Python 3 none any Details for pace_eval-0.2.2-py3-none-any.whl

Total release size: 510.8 kB

Release files / pace_eval-0.2.2.tar.gz

Download URL pace_eval-0.2.2.tar.gz
Size 245.5 kB
Tags Source
SHA-256 checksum
How to use checksums
892cd9eb57f8779757b1584c872b5357df033fbe8d83fa5c86c575dc58b53075
BLAKE2b-256 checksum
How to use checksums
59e07ca1763e3018a78c0272556029c947da3408157b9def3fdc384b0405e423
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Release files / pace_eval-0.2.2-py3-none-any.whl

Download URL pace_eval-0.2.2-py3-none-any.whl
Size 265.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9018dbb6830f7fcb80c9c229649e757d108b7669819aa41efb7b0c7c5fbf8369
BLAKE2b-256 checksum
How to use checksums
ea79395245c1b4ff63201029bd5a71a144c29a097d580f7d35141a5f23d5530d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Security for pace-eval 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.2 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page