trialdesignbench 0.2.1

Evaluate whether AI agents can create efficient, reproducible, and safe clinical trial design

trialdesignbench

TrialDesignBench provides tooling for evaluating whether AI agents can reproduce clinical trial designs from Statistical Analysis Plans and protocols.

This baseline implements workflow step 1:

1. Create a local benchmark workspace.
2. Convert a SAP/protocol PDF to Mathpix Markdown, with optional LaTeX ZIP output.
3. Build the standard TrialDesignBench reproduction prompt.
4. Run the prompt against a locally installed Codex SDK/runtime and save the run artifacts.

Installation

uv add trialdesignbench

For development:

git clone https://github.com/BBSW-org/TrialDesignBench.git
cd TrialDesignBench
uv sync

The experimental Codex Python SDK is declared as a Git source dependency for uv environments until it is published on PyPI. From a clone of this repository, uv sync installs both openai-codex and its pinned local runtime. For PyPI-only installs before openai-codex is published on PyPI, add the SDK source explicitly in the consuming project:

uv add "openai-codex @ git+https://github.com/openai/codex.git#subdirectory=sdk/python"

Quick Start

uv run tdb init tdb-workspace
uv run tdb configure --workspace tdb-workspace
uv run tdb run path/to/sap.pdf --workspace tdb-workspace --case-id tdb-001

Use --no-codex to exercise only the Mathpix ingestion portion:

uv run tdb run path/to/sap.pdf --workspace tdb-workspace --no-codex

The workspace .env file stores MATHPIX_APP_ID, MATHPIX_APP_KEY, CODEX_MODEL, and optionally CODEX_BIN. The default Codex model is gpt-5.5, and the default reasoning effort is high. The generated workspace .gitignore excludes credentials and output artifacts by default.

Converted Mathpix artifacts are reused by default on retry. Add --force to submit the PDF again, or --http-timeout to increase the per-request Mathpix HTTP timeout for large uploads.