design-research-experiments

design-research-experiments is the study-design and orchestration layer in the CMU Design Research Collective design-research ecosystem. It makes hypotheses, factors, conditions, replications, and artifact flows explicit before execution.

It composes sibling libraries rather than reimplementing them:

• design-research-agents for executable agent behavior, workflows, and traces
• design-research-problems for problem catalogs, registries, and evaluators
• design-research-analysis for downstream unified-table analysis and reporting

Quality Signals

• Coverage reports total line coverage for the default deterministic test suite; CI requires at least 95%.
• Examples Passing reports checked-in example scripts that execute successfully in the examples workflow.
• API in Examples reports curated top-level __all__ exports referenced by runnable examples. N/N means every supported top-level export appears in at least one example, and CI requires 100%.

Run make coverage, make examples-test, and make examples-coverage to reproduce these checks locally.

Overview

This package centers on reproducible experiment structure and execution:

• typed schemas for studies, factors, blocks, hypotheses, outcomes, and analysis plans
• design-of-experiments materialization (full/constrained factorial, randomized block, repeated measures, latin square, custom matrices)
• run orchestration with deterministic seeding, always-on per-run evidence, optional checkpointing, resume support, and interactive tqdm progress
• explicit, disabled-by-default paper-support aggregation and paper-draft assembly with provenance, curated references, visible TODOs, and author-review boundaries
• standalone condition callbacks for simulations that do not require synthetic problem or agent bindings
• canonical artifact exports (study.yaml, manifest.json, conditions.csv, runs.csv, events.csv, evaluations.csv, and machine-readable hypothesis/plan files)
• documented artifact contracts that downstream analysis can ingest directory-first
• documented composition seams for user code: design_research_problems.integration, the stable design_research_agents.study facade, and the top-level design_research_analysis artifact API

design-research-experiments itself is the orchestration surface. There is intentionally no separate design_research_experiments.integration module. Internally, the package adapters consume design_research_problems.integration and design_research_agents.integration; the latter is a compatibility seam, not the recommended authoring API. New user code should use design_research_agents.study.

Quickstart

Requires Python 3.12+. Maintainer workflows target Python 3.12 (.python-version). For a VS Code path that starts from PyPI and then shows the repository example workflow, see VS Code Start.

Install the base package from PyPI:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install design-research-experiments

The base install is enough to define, validate, and materialize a study:

from design_research_experiments import build_design, build_prompt_framing_study

study = build_prompt_framing_study()
conditions = build_design(study)
print(study.study_id, len(conditions))

For contributor workflows from a source checkout:

python -m venv .venv
source .venv/bin/activate
make dev
make test

Run the repository's basic example:

make run-example

This repo maintains a hard 95% total line-coverage floor in CI via make coverage. The repo-specific rule tracks the family-wide coverage policy in cmudrc/design-research#4.

CLI

The package installs a drexp CLI:

drexp validate-study path/to/study.yaml
drexp materialize-design path/to/study.yaml
drexp generate-doe --kind lhs --factors-json '{"x": [0, 1], "y": [10, 20]}' --n-samples 12 --out artifacts/doe.csv
drexp run-study path/to/study.yaml
drexp resume-study path/to/study.yaml
drexp export-analysis path/to/study.yaml
drexp draft-paper path/to/output_dir --require-complete
drexp bundle-results path/to/output_dir

Install design-research-experiments[doe] to enable optional SciPy/QMC and pyDOE3 backends for generate-doe --backend ...; the stdlib backend remains the default.

Examples

See examples/README.md for runnable scripts, including end-to-end recipe executions.

Docs

See the published documentation for guides, the artifact contract, and API reference.

For whole-stack orientation and canonical cross-package examples, use the design-research umbrella documentation.

Build docs locally with:

make docs-check
make docs-build

Public API

Selected primary entry points include:

• Study, Factor, Level, Constraint, Condition, Block, DesignSpec, DesignKind
• RecipeStudyConfig, ComparisonStudyConfig, and recipe-specific typed config classes
• Hypothesis, OutcomeSpec, AnalysisPlan
• RunSpec, RunResult, RunOutput, ConditionRunner, BenchmarkBundle
• build_design, generate_doe, materialize_conditions
• build_univariate_comparison_study, build_bivariate_comparison_study, build_strategy_comparison_study, and other recipe builders
• run_study, resume_study, load_run_evidence_records
• collect_paper_support, export_paper_support, export_paper_draft, PaperDraftIncompleteError
• export_analysis_tables, validate_study

Contributing

Contribution workflow and quality gates are documented in CONTRIBUTING.md.