PACE: Parity-Aware Compute Evaluation
More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs
Author: Seyed Mahmoud Sajjadi Mohammadabadi
Affiliation: Department of Computer Science and Engineering, University of Nevada, Reno
Correspondence: mahmoud.sajjadi@unr.edu
Repository: https://github.com/mahmoudsajjadi/pace
arXiv Preprint: arXiv:2611.17610 (HTML version)
Paper PDF: Preprint: manuscript/main.pdf | Visual Cheatsheet: docs/assets/pace_cheatsheet.pdf | Interactive Studio: dist/hf_space_pace_studio/
Reproducible on one consumer GPU. The controlled experiments (llama3.1:8b served by Ollama, about 4.9 GB) run on a single 12 GB card (developed on an RTX 4070): bash scripts/reproduce_12gb_gpu.sh runs the tests, the hidden-profile file check and a 170-call smoke run, and records its own wall time (the expected runtimes in the script header are derived from a measured batch rate, not from a timed run of the script).
Official replication repository, audited contrast ledger (9 primary studies, 516 unique multi-agent configurations from 531 raw extracted rows, 834 contrast records), pace CLI and evaluation toolkit (legacy aliases: cme, mae-eval), and zero-dependency browser workbench (PACE Studio) for Parity-Aware Compute Evaluation.
Licences. Code (code/, analysis/, scripts/, tests/): Apache-2.0 (LICENSE). Dataset files (extracted contrast tables and the hidden-profile task, staged in zenodo_datasets.json): CC-BY-4.0.
Abstract & Canonical Findings
Multi-agent language model evaluations often compare communicating systems against a single greedy pass, confounding inter-agent coordination with independent sampling and expanded test-time compute. We introduce Parity-Aware Compute Evaluation (PACE), which classifies single-agent baselines on a four-category ladder (Category A: single greedy pass $k=1$; Category B: noncommunicating sampled baseline; Category C: resource-matched or shared-ceiling baseline; Category D: compute-cost Pareto frontier), audits a resource vector with six dimensions
$$\mathbf{R} = \langle R_{\text{reason}}, R_{\text{token}}, R_{\text{cost}}, R_{\text{lat}}, R_{\text{sample}}, R_{\text{sys}} \rangle,$$
enforces directional resource feasibility, and measures the frontier gap relative to single-agent self-consistency at parity in tokens.
Extracted Corpus & Empirical Synthesis
- Canonical Denominators: From
12studies considered across two stages (5initial studies and7screened candidates:4included,3excluded), the included corpus comprises9primary studies,516unique multi-agent configurations (531raw extracted rows minus15baseline-only rows), and834baseline-specific contrast records (358Category A,360Category B,114Category C, and2Category D Pareto contrasts;474combined Category B/C contrasts across8studies;156shared-ceiling contrasts;318paired Category A/B contrasts across72unique baseline cells). - Single-Pass vs. Compute-Controlled Categories: Mean accuracy differences are
-0.032against Category A (358contrasts;-0.005omitting Zhang et al., who supply180configurations),-0.051against Category B (360contrasts), and-0.012against Category C (114contrasts). Across combined Category B/C (474contrasts,8studies), the unweighted contrast mean is-0.042, with exploratory three-level REML log odds $\hat\beta = \mathbf{-0.195}$ (CR2 95% CI[-0.323, -0.067]; 95% study-mean prediction interval[-0.515, +0.125]). - Consistency Across Studies: Multi-agent systems fail to lead in
74%(352of474) of Category B/C contrasts, feasible baselines match or beat them in36%(171of474directionally feasible;56below-1.96 SE;69%or356of516unique configurations across Categories A/B/C), and8 of 9study means are nonpositive (sole positive exception: Wunderlich et al. at+0.020at stated configurations on MMLU-Pro, compute not matched). - Sampling Gap ($\Delta_{\text{gap}}$): Across
5dual-baseline studies (318pairs spanning72unique baseline cells,64 of 72negative), switching from greedy to sampled baselines lowers accuracy differences by-0.038weighted equally by study (-0.028across pairs;-0.036across the72unique cells; cell-level REML $\hat\beta = \mathbf{-0.178}$, CR2 95% CI[-0.347, -0.009]). - Controlled Open-Weight Experiment (
llama3.1:8bvia Ollama):- Shared-evidence MMLU-Pro (
n=250): Vanilla debate (−0.015frontier gap,0.567vs.0.581frontier) and Mixture-of-Agents (−0.072frontier gap) do not clear the equal-token self-consistency frontier under Holm correction. - Complementary Hidden Profile split-view control (
n=200): Split-view debate exceeds the split-view independent-voting frontier (+0.073frontier gap,0.182vs.0.109, Holm-adjustedp = 0.007), while trailing content-free placebo prompts (0.248) and four-choice chance (0.250).
- Shared-evidence MMLU-Pro (
Repository Structure
├── .github/ # CI workflows & study-contrast issue template (add_study_contrast.yml)
├── analysis/ # Synthesis, REML pooling, plotting, and LaTeX build gates
│ ├── assemble_paper.py # Assembles & compiles manuscript/main.pdf
│ ├── run_baseline_tiers.py # Descriptive corpus synthesis, dual-baseline pairing & numbers.tex
│ └── run_k9_reml.py # Three-level REML + CR2 inference (including 72-cell Model D)
├── code/
│ ├── pace/ # Standalone PACE CLI, importers (incl. Inspect AI), & browser Studio
│ ├── cme/ # Legacy CLI/API aliases preserved for backwards compatibility
│ └── mae_eval/ # Legacy resource tracker/auditor package aliases
├── dist/
│ ├── assets/ # Explainer GIF & cheatsheet (PNG/PDF)
│ ├── github_release_staging/# Offline-verified GitHub release staging tree
│ └── hf_space_pace_studio/ # Hugging Face Space workbench + corpus_contrasts.json (832 rows)
├── docs/ # Project landing page (index.html), community playbook & visual assets
├── evidence/ # Verified extracted CSVs, screening ledger & audit reports
│ ├── contrasts_anchor5_core.csv
│ ├── contrasts_dual_baseline.csv
│ ├── extension_contrasts.csv
│ └── verification_worksheet.csv
├── figures/ # Publication vector PDF figures and PNG previews
├── manuscript/ # Modular LaTeX source of truth and generated macro files
├── notebooks/ # Interactive Colab/Jupyter log-auditing notebook (audit_your_log.ipynb)
├── outputs/ # Baseline contrast tables, REML JSON & experimental logs
├── paper/ # Synchronized manuscript build targets and figure/table mirrors
├── pyproject.toml # PEP 621 package metadata (`pace-eval`) & CLI entry points
├── review/ # Programmatic concordance ledgers & claim-verification tables
├── scripts/ # Reproducibility runner (12 GB GPU), corpus builders & exporters
├── supplement/ # Moderator synthesis, empirical trace harvest & author templates
└── tests/ # Automated unit, regression, and concordance test suite
Quickstart: the pace CLI & PACE Studio
Install pace-eval from the repository root (pyproject.toml) or from code/:
pip install -e . # installs `pace` (plus legacy `cme` and `mae-eval` console aliases)
Core auditing, reporting, log import (including Inspect AI via --format inspect), and local GUI commands:
pace studio # launches the zero-dependency browser UI (PACE Studio)
pace classify --config sas.json
pace audit run_trace.jsonl
pace import-logs --format inspect --input eval_log.json --output trace.jsonl && pace audit trace.jsonl
pace report --pool pool.jsonl --logs run.jsonl --system debate --export-latex card.tex
pace dynamics debate_log.jsonl --null null.jsonl --json
pace card run_trace.jsonl --latex --output pace-card.tex
Constructive planning, simulation, meta-analysis, frontier, and rebuttal commands:
pace pool-size --agents 5 --rounds 3 --prompt-tokens 100 --msg-tokens 20 --json
pace simulate --mode debate --items 100 --agents 5 --rounds 3 --out debate.jsonl
pace meta --contrasts outputs/baseline_tier_contrasts.csv --scale log_or --json
pace power --p-base .55 --delta .10 --n-items 40 --json
pace benchmark-gen --route clue_grid --out clues.jsonl
pace aggregate-open --input candidates.jsonl --metric token_jaccard --out centroids.jsonl
pace plot --kind phase --out phase.svg
pace hidden-profile --n 100 --out hp.jsonl
pace frontier --samples pool.jsonl --logs run.jsonl --out frontier.pdf
pace rebuttal --contrasts data/baseline_tier_contrasts.csv --export-markdown rebuttal.md
End-to-end synthetic walkthrough, interactive Colab notebook, GitHub Action CI workflow template, and community study-submission playbook live in:
notebooks/audit_your_log.ipynbcode/examples/end_to_end_walkthrough.pycode/examples/github_action_pace_audit.ymldocs/index.html,docs/COMMUNITY_AND_PR_PLAYBOOK.md, and.github/ISSUE_TEMPLATE/add_study_contrast.yml- Visual assets:
docs/assets/pace_explainer.gif,docs/assets/pace_cheatsheet.png, anddist/assets/
Legacy imports (from cme ... and from mae_eval ...) remain supported alongside pace.
Reporting Standard: Evaluation Card & Progressive Levels
Every completed comparison should report:
- The seven-field PACE Evaluation Card: Baseline rung (
A–D), resource vector $\mathbf{R}$, token accounting (prompt, completion, cached vs. uncached tokens), MAS/SAS resource ratios, frontier gap with 95% paired bootstrap CI, evidence/route structure (sharedvs.split, verifier availability, model heterogeneity), and per-item resource distribution and stopping rules. - Three progressive reporting levels:
PACE-Minimum: Category B/C baseline (e.g., nested initial-round self-consistency voting), full token/call counts, and paired bootstrap accuracy intervals.PACE-Recommended: Equal-token self-consistency frontier curve, directional feasibility across all six dimensions of $\mathbf{R}$, and placebo control or split-view checks.PACE-Extended: Per-item budget matching ($k_i$), cache-weighted billing accounting, adaptive stopping distributions, hybrid call-graph edge counts, and peer-influence transition dynamics ($\tau_{i\to c}$ vs. $\tau_{c\to i}$).
Verification & Manuscript Compilation
Run the full automated verification suite (865 passed, 8 skipped):
pytest -q tests/
Assemble and verify all PDF targets (manuscript/main.pdf, paper/main.pdf) with Build Gates 1 and 2:
python -I analysis/assemble_paper.py
Citation
@article{sajjadi2026more,
title = {More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs},
author = {Sajjadi Mohammadabadi, Seyed Mahmoud},
journal = {arXiv preprint arXiv:2611.17610},
year = {2026},
eprint = {2611.17610},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
doi = {10.48550/arXiv.2611.17610},
url = {https://arxiv.org/abs/2611.17610},
note = {Replication repository at https://github.com/mahmoudsajjadi/pace}
}
Metadata
Release files for pace-eval 0.2.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pace_eval-0.2.2.tar.gz | 245.5 kB | Without provenance | Details for pace_eval-0.2.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pace_eval-0.2.2-py3-none-any.whl | Python 3 | none | any | Details for pace_eval-0.2.2-py3-none-any.whl |
Total release size: 510.8 kB
Release files / pace_eval-0.2.2.tar.gz
| Download URL | pace_eval-0.2.2.tar.gz |
|---|---|
| Size | 245.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
892cd9eb57f8779757b1584c872b5357df033fbe8d83fa5c86c575dc58b53075
|
|
BLAKE2b-256 checksum How to use checksums |
59e07ca1763e3018a78c0272556029c947da3408157b9def3fdc384b0405e423
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.10
|
Release files / pace_eval-0.2.2-py3-none-any.whl
| Download URL | pace_eval-0.2.2-py3-none-any.whl |
|---|---|
| Size | 265.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9018dbb6830f7fcb80c9c229649e757d108b7669819aa41efb7b0c7c5fbf8369
|
|
BLAKE2b-256 checksum How to use checksums |
ea79395245c1b4ff63201029bd5a71a144c29a097d580f7d35141a5f23d5530d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.10
|
Security for pace-eval 0.2.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.