orcca 0.1.1

Manifest-driven clinical workflow orchestration framework

Orcca

Orcca is a manifest-driven orchestration framework for clinical programming workflows. It is designed for teams that need reproducible execution, explicit dependency modeling, and clear run reporting.

Current State

• Package version: 0.1.0 (alpha)
• Python: >=3.11
• Core dependencies: PyYAML, jsonschema, linkml
• Entry point: orcca CLI (orcca.cli:main)

This project is currently optimized for practical workflow execution and validation with a strong traceability model, not for broad general-purpose scheduling.

What Orcca Does

Orcca provides:

• declarative workflow modeling in YAML
• manifest include/config resolution
• LinkML + business-rule validation
• executable task orchestration with dependency-aware scheduling
• runtime preflight for package-manager and container environments
• run reporting (run_report.html) and task logs
• validation task execution for independent double programming (IDP)
• blinding_mode-based blinded/unblinded workflow control

Core Architecture

Main modules:

• orcca/resolver.py: include/config merge and placeholder interpolation
• orcca/validator.py: LinkML validation + business rules
• orcca/executor.py: task execution engine + runtime preflight
• orcca/workflow_runner.py: end-to-end orchestration pipeline and run reports
• orcca/runtime_launchers.py: built-in + custom launcher resolution
• orcca/dagger.py: generated-artifact DAG utilities
• orcca/cli.py: user-facing commands

Packaged schema/rules:

• orcca/resources/schema/flow-linkml.yaml
• orcca/resources/rules/business-rules.yaml

Manifest Resolution Semantics

Orcca resolves manifests in this order:

1. Resolve includes recursively in depth-first, declaration order.
2. Merge include payloads into a single root payload.
3. Merge the root manifest payload on top.
4. Resolve configs in declaration order.
5. Interpolate ${config['output_dir']}-style placeholders after merging is complete.
6. Placeholder substitutions must resolve to scalar values, not lists or dictionaries.

Merge behavior:

• dict values replace completely
• list-valued keys inside dicts: merged with the same list rules
• lists of scalars: append + dedupe while preserving first-seen order
• lists of {id: ...} entities: overwrite by id
• scalars/type mismatch: latest wins

Config files use the same merge rules as manifest root slots, so later files can override earlier ones without changing the merge model.

Declarative slots such as id, name, description, notes, envvar_name, and param_name must be written as explicit literals and may not use ${config['output_dir']}-style interpolation. Placeholder substitutions must resolve to scalar values, not lists or dictionaries.

This matters for override design: unblinded manifests should include base first, then override files that only change allowed slots.

Workflow Model

Primary slots:

• study
• standards
• artifacts
• artifact_groups
• runtimes
• tasks
• validations
• deliveries
• execution
• blinding_mode

Important modeling rules:

• tasks.outputs must reference declared artifacts.
• tasks.inputs can reference declared artifacts or transient IDs used internally.
• required_tasks provides explicit execution sequencing independent of artifact lineage.

Blinded/Unblinded Mode

Run mode:

blinding_mode: blinded # or unblinded

Artifact variants:

• variant is optional and only needed for paired blinded/unblinded artifacts.

artifacts:
  - id: rand_map
    variant: blinded
    name: Randomization Map
    type: lookup_dataset
    origin: external
    path: outputs/rand_map_blinded.csv

Unblinded pattern:

includes:
  - workflow_blinded.yaml
  - manifests/artifacts_unblinded.yaml

blinding_mode: unblinded

Declarative slots must remain literal and may not use quoted config interpolation placeholders.

CLI

Top-level commands:

• orcca validate <manifest.yaml>
• orcca plan <manifest.yaml>
• orcca run <manifest.yaml> [--project-root <path>]
• orcca env-cache list|prune [--all]
• orcca init [path]

Useful run flags:

• --task <id> (repeatable)
• --artifact <id> (repeatable)
• --run-validations
• --validation-id <id> (repeatable; requires --run-validations)
• --no-execute
• --no-env-cache
• --refresh-env

Run root behavior:

• --project-root controls execution root and run artifact root
• runs are stored under <project-root>/.orcca/runs

Execution and Reports

run_workflow(...) pipeline:

1. resolve manifest
2. validate manifest
3. execute tasks (optional)
4. package deliveries (when configured)
5. optionally execute selected validation tasks
6. write run artifacts

Per-run artifacts:

• resolved_manifest.yaml
• run_report.html
• task logs under logs/
• task context payloads under task-context/

Runtime Model

Built-in launchers:

• python
• rscript
• julia
• container

Custom launchers:

• auto-loaded from launchers/*.py
• optional launchers/register.py
• launcher file rule: filename stem must match exported function name

Package-manager environments:

• cached under .orcca/cache/environments/
• managers: pip, venv, renv, julia_pkg (+ explicit restore command for others)

Container environments:

• source types: registry, archive, build
• archive load and local build workflows supported

Validation and IDP

Validation uses:

• LinkML schema validation
• business-rule validation
• cross-reference checks and dependency checks

IDP (check_type: independent_double_programming) supports:

• artifact/group targets
• explicit validation_task
• validation targets and validation task references
• tool metadata via tool

Recommended IDP pattern:

• keep targets for business ownership
• use validation_task for executable provenance
• keep validation outputs easy to inspect in reports and logs

Examples in Repo

Reference workflows:

• testing/e2e_python_native/workflow.yaml
• testing/e2e_r_native/workflow.yaml
• testing/e2e_julia_native/workflow.yaml
• testing/e2e_sdtm/workflow.yaml

Blinding handoff example:

• testing/blinding_handoff/README.md

Development

Install:

pip install -e .[dev]

Run tests:

pytest -q