design-research-experiments

design-research-experiments is the hypothesis-first study-definition and experiment-orchestration layer in the cmudrc design research ecosystem.

It composes sibling libraries rather than reimplementing them:

• design-research-agents for executable agent behavior, workflows, and traces
• design-research-problems for problem catalogs, registries, and evaluators
• design-research-analysis for downstream unified-table analysis and reporting

Quality Signals

• Coverage reports total line coverage for the default deterministic test suite; CI requires at least 95%.
• Examples Passing reports checked-in example scripts that execute successfully in the examples workflow.
• API in Examples reports curated top-level __all__ exports referenced by runnable examples. N/N means every supported top-level export appears in at least one example, and CI requires 100%.

Run make coverage, make examples-test, and make examples-coverage to reproduce these checks locally.

Overview

This package centers on reproducible experiment structure and execution:

• typed schemas for studies, factors, blocks, hypotheses, outcomes, and analysis plans
• design-of-experiments materialization (full/constrained factorial, randomized block, repeated measures, latin square, custom matrices)
• run orchestration with deterministic seeding, checkpointing, resume support, and interactive tqdm progress on terminal runs
• standalone condition callbacks for simulations that do not require synthetic problem or agent bindings
• canonical artifact exports (study.yaml, manifest.json, conditions.csv, runs.csv, events.csv, evaluations.csv, and machine-readable hypothesis/plan files)
• documented artifact contracts that downstream analysis can ingest directory-first
• thin orchestration adapters that delegate sibling-package interoperability to design_research_problems.integration, design_research_agents.integration, and the top-level design_research_analysis artifact API

design-research-experiments itself is the orchestration surface. There is intentionally no separate design_research_experiments.integration module.

Quickstart

Requires Python 3.12+. Maintainer workflows target Python 3.12 (.python-version). For a VS Code path that starts from PyPI and then shows the repository example workflow, see VS Code Start.

python -m venv .venv
source .venv/bin/activate
make dev
make test

Run a basic example:

make run-example

This repo maintains a hard 95% total line-coverage floor in CI via make coverage. The repo-specific rule tracks the family-wide coverage policy in cmudrc/design-research#4.

CLI

The package installs a drexp CLI:

drexp validate-study path/to/study.yaml
drexp materialize-design path/to/study.yaml
drexp generate-doe --kind lhs --factors-json '{"x": [0, 1], "y": [10, 20]}' --n-samples 12 --out artifacts/doe.csv
drexp run-study path/to/study.yaml
drexp resume-study path/to/study.yaml
drexp export-analysis path/to/study.yaml
drexp bundle-results path/to/output_dir

Install design-research-experiments[doe] to enable optional SciPy/QMC and pyDOE3 backends for generate-doe --backend ...; the stdlib backend remains the default.

Examples

See examples/README.md for runnable scripts, including end-to-end recipe executions.

Docs

See the published documentation for guides, the artifact contract, and API reference.

Build docs locally with:

make docs

Public API

Top-level exports are intentionally small:

• Study, Factor, Level, Constraint, Condition, Block, DesignSpec, DesignKind
• RecipeStudyConfig, ComparisonStudyConfig, and recipe-specific typed config classes
• Hypothesis, OutcomeSpec, AnalysisPlan
• RunSpec, RunResult, RunOutput, ConditionRunner, BenchmarkBundle
• build_design, generate_doe, materialize_conditions
• build_univariate_comparison_study, build_bivariate_comparison_study, build_strategy_comparison_study, and other recipe builders
• run_study, resume_study
• export_analysis_tables, validate_study

Contributing

Contribution workflow and quality gates are documented in CONTRIBUTING.md.