trade-study

Multi-objective trade-study orchestration: define factors, build parameter grids, run hierarchical study phases, and extract Pareto fronts — for any domain where you compare alternatives against competing objectives.

Statement of need

Comparing design alternatives against multiple objectives is a common task across scientific simulation, engineering trade-offs, and ML hyperparameter tuning. Researchers typically glue together separate tools for grid construction, execution, scoring, and Pareto analysis, writing ad-hoc scripts that are hard to reproduce or extend to multi-phase studies (screening → refinement → benchmark).

trade-study provides a single orchestration layer that composes these steps into a reproducible, protocol-driven workflow. Users supply a Simulator (generates data) and a Scorer (evaluates it); the framework handles grid construction, parallel execution, Pareto extraction, and phase chaining. All components are modular and optional — use only what you need.

This package targets researchers and practitioners who need:

• structured multi-phase experimental design (screen → refine → benchmark),
• multi-objective Pareto analysis across heterogeneous factors, and
• a reproducible Python API that separates domain logic from study orchestration.

Why trade-study?

Existing tools solve pieces of this problem — optuna for adaptive optimization, pymoo for multi-objective solvers, SALib for sensitivity analysis — but none provide the hierarchical phase orchestration that connects them into a single study.

Quick start

This complete example needs NumPy and the pareto extra: pip install 'trade-study[pareto]'. Each replicate uses the same sample bank for both designs, so errors can be compared in pairs.

import numpy as np

from trade_study import (
    Direction, Factor, FactorType, Observable, build_grid,
    extract_front, paired_difference, run_grid,
)


class SampleMean:
    def generate(self, config, *, rep=0):
        rng = np.random.default_rng(np.random.SeedSequence([2026, rep]))
        return 0.0, rng.normal(size=32)[:config["samples"]]


class ErrorAndCost:
    def score(self, truth, observed, config):
        return {"error": float((observed.mean() - truth)**2),
                "cost": float(config["samples"])}


grid = build_grid(
    [Factor("samples", FactorType.DISCRETE, levels=[8, 32])], method="full"
)
observables = [Observable("error", Direction.MINIMIZE),
               Observable("cost", Direction.MINIMIZE)]
results = run_grid(SampleMean(), ErrorAndCost(), grid, observables, n_reps=64)
means = results.aggregate_replicates()
front = extract_front(means.scores, [o.direction for o in observables])
contrast = paired_difference(results, {"samples": 32}, {"samples": 8},
                             "error", seed=2026)
print(means.scores, front, contrast.mean, contrast.n_pairs)

results retains all 128 trial rows; means has one row per design. Bootstrap resamples in the paired interval do not increase the number of independent replicates. Inspect n_pairs: this helper excludes nonfinite pairs. A study must define its failure policy before interpreting a success-only comparison. Positive objective scaling preserves Pareto membership; use decision preferences when choosing among nondominated designs.

The typed standalone version is replicated_study.py. The capabilities and example checks map additional APIs, optional dependencies, and executable examples.

For a complete 15-minute demonstration with live code and saved figures, see the serosurvey design notebook and presentation guide. From a repository checkout, launch it with uv run --extra notebook jupyter lab examples/serosurvey_study.ipynb.

Protocols

Users implement two protocols to plug in their domain:

from trade_study import Scorer, Simulator


class MySimulator:
    """Implements the Simulator protocol."""

    def generate(self, config: dict) -> tuple:
        """Return (truth, observations) for a given config."""
        ...


class MyScorer:
    """Implements the Scorer protocol."""

    def score(self, truth, observations, config: dict) -> dict[str, float]:
        """Return {observable_name: value} for a single trial."""
        ...

Installation

pip install trade-study[all]

Or install only the extras you need:

pip install trade-study[design,pareto]

Core dependency: numpy only.

API overview

The snippets below show API composition; names such as world, scorer, run_fn and results stand for caller-supplied objects. Use the complete quickstart above or a worked guide to run a study.

Design

from trade_study import Factor, FactorType, build_grid, screen

factors = [
    Factor("x", FactorType.CONTINUOUS, bounds=(0.0, 1.0)),
    Factor("method", FactorType.CATEGORICAL, levels=["a", "b", "c"]),
]

grid = build_grid(factors, method="sobol", n_samples=1024)
si = screen(run_fn, factors, method="morris", n_trajectories=3000)

Execution

from trade_study import run_grid, run_adaptive

# Grid mode: evaluate all configs (optional parallelism)
results = run_grid(world, scorer, grid, observables, n_jobs=-1)

# Adaptive mode: evolutionary multi-objective optimization (Optuna NSGA-II)
results = run_adaptive(world, scorer, factors, observables, n_trials=600)

Pareto analysis

from trade_study import extract_front, hypervolume, pareto_rank

front_idx = extract_front(results.scores, directions)
ranks = pareto_rank(results.scores, directions)
hv = hypervolume(results.scores[front_idx], ref_point, directions)

Stacking

from trade_study import stack_scores, stack_bayesian, stack_proportional, ensemble_predict

weights = stack_scores(score_matrix)  # select best means; split exact ties
weights = stack_proportional(score_matrix)  # preserve weight across near ties
weights = stack_bayesian(idata_dict)  # arviz stacking (Yao et al. 2018)
combined = ensemble_predict(predictions, weights)

stack_scores solves the supplied linear-score objective directly using NumPy; positive changes of score units preserve its optimum. ensemble_predict normalizes a copy of the weights and accepts read-only arrays and views. See choosing ensemble weights for examples and the distinction between input scores and the score of a combined prediction.

I/O

from trade_study import save_results, load_results

save_results(results, "study_results")
results = load_results("study_results")
frame = results.to_dataframe()  # optional pandas extra; includes trial metadata
frame.to_csv("study_results.csv", index=False)

Multi-fidelity search and surrogates

from trade_study import run_successive_halving, run_hyperband, fit_surrogate, fit_regime_surrogate

# Cheap-first budget-constrained screening
results = run_successive_halving(world, scorer, factors, observables, budget_param="maxiters")
results = run_hyperband(world, scorer, factors, observables, budget_param="maxiters")

# Interpolate scores from a completed results table (GP or RF)
surrogate = fit_surrogate(results, method="gp")

# Recommend a config per regime, interpolating over regime descriptors
regime_surrogate = fit_regime_surrogate(results, regime_cols=["n", "p"])

Surrogate fits report cross-validated cv_r2 and cv_rmse; check those diagnostics before trusting recommendations or post-hoc sensitivity results.

Constraint / feasibility_filter express infeasible regions of the design space; FactorConstraint couples factors during grid construction (e.g. keep x + y <= 1); screen() supports both Morris and Sobol sensitivity analysis. run_grid(..., n_reps=N) supports replicated trials, and sensitivity_from_table() estimates post-hoc sensitivity from an existing results table through a validated surrogate.

Release compatibility

Version 0.5.1 fixes score-unit sensitivity in stack_scores and prevents ensemble_predict from modifying supplied weights, including read-only arrays and views. It also updates executable documentation and illustrated examples.

Version 0.5.0 adds a prespecified fixed replication policy and versioned replication-contract metadata. Adaptive allocation remains the default. Pareto operations now identify the optional pareto extra when it is missing.

Version 0.4.0 adds decision reporting (preference_sweep() with regret, practical equivalence and optional paired uncertainty), fixed-sample proportion intervals and replicate sizing, versioned result bundles with provenance, sequential replication (run_sequential(), with explicit fixed/adaptive policy and recorded counts/uncertainty assumptions), evaluation caching, incremental grid checkpoints with bounded retries, grouped surrogate validation with support diagnostics, and the IVAC serosurvey notebook. The adaptive extra now requires Optuna 4.5 or later, and strict </> adaptive constraints treat exact equality as infeasible, matching result-table feasibility.

Version 0.3.0 adds replicated and paired studies, uncertainty diagnostics, persistent adaptive sessions, checkpointing, and DataFrame export. Existing phase-name-only checkpoints are refused; use a fresh checkpoint directory and set checkpoint_key to track model/data revisions. Confidence constraints require estimated standard errors rather than assuming unknown uncertainty is zero. stack_proportional() now uses exponential score utilities in both directions; set its temperature in score units to control concentration.

Related packages

Development

See CONTRIBUTING.md for setup, code standards, and the PR workflow.

uv sync --extra dev
just ci          # lint → mypy --strict → pytest with coverage
just format      # auto-format
just check       # auto-fix lint

Citation

If you use this package in your research, please cite:

@software{trade_study2026,
  author = {Macdonald, Joshua C.},
  title = {{trade-study}: Multi-Objective Trade-Study Orchestration},
  year = {2026},
  url = {https://github.com/jcm-sci/trade-study},
  version = {0.5.1},
  doi = {10.5281/zenodo.19599838},
}

See CITATION.cff for machine-readable metadata.

License

MIT