CADDForge Core

CADDForge Core provides the shared abstractions, config models, and CLI that all CADDForge plugin tools build on. It is intentionally lightweight and backend-agnostic so that tool-specific packages (e.g., cadd_qm, cadd_md, cadd_fep, cadd_descriptors) can focus on scientific logic while still benefiting from standardized execution, logging, and results handling.

This repo is designed to be open-source friendly: a clean API surface, simple configuration, and a CLI that is discoverable and reproducible.

---

Table of Contents

• Key Features
• Installation
• Quick Start
• Python API
• Pipelines
• Core Concepts
• Configuration
• CLI Reference
• Job Layout & Registry
• Backends
• Plugin Architecture
• Development & Testing
• License

---

Key Features

• Stable core abstractions: Dataset, MoleculeSet, Workflow, Job, Backend, Result.
• Config validation via Pydantic models, with YAML/JSON support.
• Backend-agnostic execution (local + Slurm built in).
• Standardized run directories, metadata, and result serialization.
• Plugin discovery through Python entry points or dev overrides.
• CLI-first UX with run, show-config, status, logs, and more.

---

Installation

Requirements: Python 3.10+

Install only core infrastructure and the CLI:

python -m pip install caddforge-core

pip install caddforge installs the complete public toolbox through a metadata-only bundle, including GUI and user-facing extras. Individual plugins install core automatically. The core distribution remains independent of scientific plugins; Python imports and the caddforge command retain their names.

For an existing environment using the former caddforge core distribution, prefer a fresh environment. For in-place migration, uninstall caddforge first, then install caddforge-core and upgrade all retained plugins to their migrated releases (or install the complete bundle). The old and renamed core distributions own the same Python files and must not coexist. If both were already installed, uninstall the old distribution and force-reinstall core.

Runtime version queries now use importlib.metadata.version("caddforge-core"). version("caddforge") reports the optional bundle version. Configuration formats, plugin entry-point groups, job registries, and historical results remain compatible.

From source (editable):

cd caddforge-core
python -m pip install -e .

If you are using multiple tool plugins, install them in the same environment so the CLI can discover their entry points.

---

Quick Start

List installed tools:

caddforge list-tools

Show a tool's default config template:

caddforge show-config descriptors ani-psa

Run a tool from config:

caddforge run descriptors ani-psa --config path/to/config.yaml

Check status and logs:

caddforge status <job-id>
caddforge logs <job-id>
caddforge logs <job-id> --follow

---

Python API

Use submit_workflow when Python code should follow the same backend selection, run-directory, metadata, and registry path as the CLI:

from caddforge.core.execution import load_result_for_job, submit_workflow
from caddforge.plugins import get_tool

spec = get_tool("descriptors", "ani-psa")
cfg = spec.config_cls.from_yaml("ani_psa.yaml")
workflow = spec.workflow_cls(cfg)

job = submit_workflow(workflow, cfg, backend_name=cfg.backend)
result = load_result_for_job(job)
if result is not None:
    print(result.status.value, result.metrics)

Calling workflow.prepare_job() followed by workflow.run(job) is still useful for tests or custom inline execution, but it does not submit to a backend. Backend submission is handled by submit_workflow, submit_prepared_workflow_job, or a Backend instance such as LocalBackend / SlurmBackend.

---

Pipelines

Pipelines chain multiple CADDForge workflows through file-based artifact references. This keeps large data on disk, works with local and Slurm backends, and lets downstream stages receive paths produced by upstream stages.

Create a pipeline YAML file, for example pipeline.yaml:

version: 1
name: docking-screen

defaults:
  backend: local
  registry_path: runs/job_registry.sqlite
  job_name: unidock-screen-test

stages:
  - id: unidock_screen
    tool: docking/unidock
    config: unidock.yaml
    scatter:
      by: ligand_multisdf_path
      chunk_size: 50000
    outputs:
      scores: best_scores_csv
      poses: poses_dir

  - id: top_score_filter
    type: filter.tabular
    inputs:
      table: unidock_screen.scores
    config:
      sort_by: best_score
      ascending: true
      top_n: 1000
      output_name: filtered_scores.csv

  - id: retain_unidock_poses
    type: filter.molecules
    inputs:
      molecules: unidock_screen.poses
      ids: top_score_filter.filtered_table
    config:
      id_column: ligand

  - id: gnina_rescore
    tool: docking/gnina
    config: gnina.yaml
    inputs:
      ligand_paths: retain_unidock_poses.molecules
    outputs:
      scores: best_scores_csv
      poses: poses_dir

Run it:

caddforge pipeline run --config pipeline.yaml

Inspect the pipeline and its artifacts:

caddforge pipeline status <pipeline-id>
caddforge pipeline show-artifacts <pipeline-id>

Important details:

• tool uses the same section/name pair shown by caddforge list-tools.
• config can be an inline mapping or a YAML/JSON config file for that tool.
• defaults.job_name names the parent pipeline job. If omitted, CADDForge generates one with the same pet-name helper used by normal workflow configs.
• outputs maps pipeline artifact names to keys in the workflow Result.paths, such as best_scores_csv or poses_dir.
• Tool plugins can also declare default output ports. Optional ports are skipped when the workflow did not produce that result path, and manifest-backed ports can expose a CSV path column as a multi-path artifact for scatter.
• inputs maps downstream config fields to upstream artifacts using stage_id.artifact_name.
• scatter splits a large file input into shard jobs. Supported split strategies are sdf, csv, tsv, lines, or auto.
• Tool plugins may declare a SplitContract for structured collection fields. The contract selects the path member to split, preserves the other entry fields in every shard, and declares how shard outputs are reduced.
• scatter can also fan out a multi-path artifact, such as poses.extract.poses. If the target config field is a single Path, each path becomes one shard job. If it is list[Path], paths are chunked according to chunk_size or chunks.
• When a scattered stage receives additional path-collection inputs with the same number of paths as the scatter input, those paths are bound by row order for each shard. This supports paired files selected from the same provenance table, such as ABFE ligand_sdf and matching protein_pdb inputs. If the paired config field is typed as list[Path], each shard receives a one-item list containing the matching path, for example plain MD molecule_sdf_paths.
• In the current implementation, scatter shards are submitted as separate child jobs and reduced before downstream stages run.
• Pipeline tool stages support transparent .csv.gz, .tsv.gz, and .sdf.gz boundaries. Plugins declare gzip-capable inputs with InputPort.config_field and gzip-capable outputs by setting OutputPort(..., config_field="output_path"). Core stream-decompresses declared inputs only when the tool starts, lets the tool write a plain staging output, and atomically creates the requested gzip result. On single-node Slurm jobs, plain inputs live under $SLURM_TMPDIR; local jobs use job-scoped system temporary storage. Multi-node jobs use a private shared staging directory because one node's local scratch is not visible to the other nodes. This explicit contract prevents existing outputs, unrelated archives such as Conda .tar.gz packs, and same-named files from being misclassified.
• Original compressed paths remain in saved configs and results; runtime materializations are reused within a job and removed after it finishes. If local scratch fills, the job fails clearly instead of silently expanding the input on shared storage. Plugins mark native compressed inputs with InputCompressionPolicy.NATIVE; the Glide .sdfgz convention remains separate and unchanged.
• Plugin contracts may select nested paths with dotted collection syntax, for example InputPort("inputs", ArtifactKind.FILE, "input_files", runtime_selector="input_files[].path"). config_field remains the pipeline binding target, while runtime_selector only identifies paths to stage. The default compression policy is transparent gzip for backward compatibility. Use compression=InputCompressionPolicy.NATIVE when a tool consumes the gzip stream itself or when .gz is part of its native file format.
• Scattering a gzip source with chunk_size reads the data in one streaming pass and writes gzip shards on shared storage. Each shard job expands only its own shard in runtime scratch. Using chunks still requires a count pass to determine balanced shard sizes. Scattered stages may provide one unique gzip output path per shard; one shared output path is rejected because concurrent shard writes would corrupt it.
• When scatter fans out a multi-path artifact, CADDForge writes a scatter_manifest.csv for the scattered stage. This preserves row-level provenance such as ligand, pose_id, and pose_file so later stages can collect per-shard results and join them back to the input pose.
• When a retained artifact contains multiple paths and the downstream config field is typed as list[Path], CADDForge passes the path list. This is how retain_unidock_poses.molecules is routed into Gnina's ligand_paths.
• The example above sorts UniDock's best_scores_csv, which currently contains ligand, pose_file, and best_score. If you want ligand efficiency, add or join a descriptor column such as heavy_atom_count upstream, then use derive: {ligand_efficiency: best_score / heavy_atom_count}.

Built-in filter stages

Built-in stages use type: instead of tool:. They have fixed output artifact names, so those names do not need to be listed in an outputs: block. You reference them downstream as stage_id.output_name.

filter.tabular

Filters, ranks, and rewrites delimited tabular files. CSV and TSV inputs are supported.

Required source (choose exactly one):

• table: a tabular artifact from an upstream stage, for example unidock_screen.scores.
• input_path: a file path in config, for a source stage that reads a raw table directly. A relative path is resolved against the pipeline specification directory.

Optional inputs:

• ids: a tabular artifact containing identifiers to retain before filtering and ranking. By default, filter.tabular matches the ligand column in both tables.

Output artifacts:

• filtered_table: the filtered CSV or TSV file. Downstream reference: top_score_filter.filtered_table.

Config keys:

• input_path: optional raw CSV or TSV input. This cannot be used together with the table input.
• input_delimiter: optional one-character delimiter override. Otherwise, .csv uses a comma, .tsv and .tab use a tab, and other extensions are detected from the file contents.
• derive: optional mapping of new column names to restricted arithmetic/comparison expressions.
• filters: optional list of column filters. Supported operators are ==, !=, <, <=, >, >=, in, and not in.
• where: optional restricted boolean expression evaluated per row.
• sort_by: optional column used for sorting.
• ascending: sort direction. Set to true for ascending order or false for descending order.
• top_n: optional number of rows to retain after filtering/sorting.
• group_by: optional column used with top_n_per_group.
• top_n_per_group: optional number of rows to retain per group after filtering/sorting.
• columns: optional output column order/subset.
• id_column: column in table used with the optional ids input. Defaults to ligand.
• ids_column: column in ids used with the optional ids input. Defaults to id_column.
• output_name: output filename under the stage results directory; defaults to filtered_scores.csv. A .tsv or .tab extension selects tab-separated output; other extensions default to comma-separated output.
• output_delimiter: optional one-character output delimiter override.

Example using an upstream artifact:

- id: top_score_filter
  type: filter.tabular
  inputs:
    table: unidock_screen.scores
  config:
    sort_by: best_score
    ascending: true
    top_n: 1000

Example using a raw TSV source next to the pipeline specification:

- id: reinvent_prefilter
  type: filter.tabular
  config:
    input_path: reinvent_export.tsv
    output_name: filtered_reinvent.tsv
    filters:
      - column: Score
        op: ">="
        value: 0.75

The output is referenced as:

top_score_filter.filtered_table

poses.extract

Splits docking pose artifacts into one normalized SDF file per pose and writes a pose manifest. Use this when a docking tool produces multiple poses per ligand and downstream stages should receive individual pose files.

Required inputs:

• poses: a pose artifact, for example unidock_screen.poses.

Optional inputs:

• table: a CSV artifact with score/metadata rows, for example unidock_screen.scores.

Output artifacts:

• manifest: a CSV pose manifest. Downstream reference: extract_poses.manifest.
• poses: a multi-path artifact containing the extracted single-pose SDF files. Downstream reference: extract_poses.poses.

Manifest columns:

• ligand: ligand identifier.
• pose_id: stable pose identifier generated from ligand and rank.
• pose_rank: rank/order among poses for that ligand.
• pose_file: extracted single-pose SDF path.
• source_pose_file: original SDF/SDFGZ file.
• source_record_index: record index inside the original file.
• Any matching columns from the optional table, plus an optional SDF property selected with score_column.

Config keys:

• ligand_column: ligand column in the optional table. Defaults to ligand.
• path_column: pose path column in the optional table. If omitted, CADDForge auto-detects common path columns such as pose_file.
• score_column: optional SDF property to copy into the manifest when it is not already present in the table.
• sort_by: optional manifest column used for sorting.
• ascending: sort direction. Set to true for ascending order or false for descending order.
• top_n_per_ligand: optional number of poses to retain per ligand after sorting.
• output_name: manifest filename; defaults to pose_manifest.csv.

Example: extract the best three poses per ligand, then fan them out one pose per downstream job:

- id: extract_unidock_poses
  type: poses.extract
  inputs:
    poses: unidock_screen.poses
    table: unidock_screen.scores
  config:
    ligand_column: ligand
    score_column: dock_score
    sort_by: dock_score
    ascending: true
    top_n_per_ligand: 3

- id: downstream_per_pose
  tool: my-section/my-tool
  config: downstream.yaml
  inputs:
    ligand_path: extract_unidock_poses.poses
  scatter:
    by: ligand_path
    chunk_size: 1

molecules.hydrogenate

Adds missing explicit hydrogens to retained molecules or poses. This is useful after docking tools that remove non-polar hydrogens and before downstream tools that expect complete ligand structures. Place this stage after filtering and pose extraction so only retained poses are repaired.

Required inputs:

• molecules: molecule/pose artifact. This can be a single SDF/SDFGZ, multi-record SDF/SDFGZ, directory, or multi-path artifact such as extract_gnina_poses.poses.

Optional inputs:

• templates: fully protonated template molecules used to preserve protonation/formal charges and bond orders.
• table: CSV/manifest used to find template paths or merge metadata.

Output artifacts:

• molecules: repaired single-record SDF path collection.
• manifest: CSV with input/output paths, hydrogen counts, template status, and messages.

Config keys:

• ligand_column: ligand identifier column. Defaults to ligand.
• pose_id_column: pose identifier column. Defaults to pose_id.
• template_path_column: template path column. Defaults to auto, which detects columns such as template_file, original_ligand_file, source_molecule_file, ligand_file, or molecule_file.
• on_template_failure: warn, fail, or ignore. Defaults to warn.
• on_failure: fail or warn. Defaults to fail.
• add_coords: generate coordinates for added hydrogens. Defaults to true.
• output_name: manifest filename; defaults to hydrogenated_manifest.csv.

Template behavior:

• If a template path column is available, CADDForge loads only those template files for the retained poses.
• If no template path column is available but a templates artifact is provided, CADDForge indexes templates by SDF title and ligand ID. This is convenient for small retained sets, but explicit template paths are preferred for very large libraries.
• If no matching template is found and on_template_failure: warn, CADDForge falls back to direct RDKit hydrogen addition and records that fallback in the manifest.

Example:

- id: repair_gnina_pose_hydrogens
  type: molecules.hydrogenate
  inputs:
    molecules: extract_gnina_poses.poses
    templates: retain_unidock_poses.molecules
  config:
    ligand_column: ligand
    on_template_failure: warn
    on_failure: fail

- id: openbpmd
  tool: md/openbpmd
  config: openbpmd.yaml
  inputs:
    ligand_path: repair_gnina_pose_hydrogens.molecules
  scatter:
    by: ligand_path
    chunk_size: 1

results.collect

Collects JSON result files from a scattered stage into one CSV table and joins each result back to the scattered input provenance. Use this after per-pose tools such as OpenBPMD when you need to rank or filter poses by a result metric.

Required inputs:

• results: a result artifact from a scattered tool stage, for example openbpmd.results.

Optional inputs:

• table: provenance CSV. If omitted, CADDForge uses the scattered stage's automatically generated scatter_manifest.csv.

Output artifacts:

• table: collected CSV table. Downstream reference: collect_openbpmd.table.

Config keys:

• metrics: mapping of output CSV column names to JSON paths inside each result record. For a standard CADDForge Result, use paths such as metrics.comp_score.
• output_name: output filename; defaults to collected_results.csv.

Behavior:

• Rows are matched to scattered inputs by shard order. For path-collection scatter, the automatic provenance includes pose metadata from upstream manifests, commonly ligand, pose_id, pose_rank, and pose_file.
• If a tool writes both a human-readable results.json and a canonical result_record.json in the same results directory, CADDForge reads the canonical record for metric extraction.

Example: keep the most stable OpenBPMD pose per ligand, where lower comp_score is better:

- id: openbpmd
  tool: md/openbpmd
  config: openbpmd.yaml
  inputs:
    ligand_path: repair_gnina_pose_hydrogens.molecules
  scatter:
    by: ligand_path
    chunk_size: 1
  outputs:
    results: results_json

- id: collect_openbpmd
  type: results.collect
  inputs:
    results: openbpmd.results
  config:
    metrics:
      comp_score: metrics.comp_score

- id: most_stable_pose_per_ligand
  type: filter.tabular
  inputs:
    table: collect_openbpmd.table
  config:
    sort_by: comp_score
    ascending: true
    group_by: ligand
    top_n_per_group: 1
    output_name: most_stable_openbpmd_poses.csv

- id: retain_stable_openbpmd_poses
  type: filter.molecules
  inputs:
    molecules: repair_gnina_pose_hydrogens.molecules
    ids: most_stable_pose_per_ligand.filtered_table
  config:
    id_column: pose_id
    path_column: output_file

filter.molecules

Retains molecule or pose files whose identifiers occur in a filtered table.

Required inputs:

• molecules: a molecule/pose artifact. This can be a file, a directory, or a multi-path artifact.
• ids: a CSV artifact containing identifiers to retain, usually from filter.tabular.

Output artifacts:

• molecules: the retained molecule/pose artifact. Downstream reference: retain_unidock_poses.molecules.

Config keys:

• id_column: column in the ids table containing identifiers. Defaults to ligand_id.
• path_column: optional column containing exact molecule/pose file paths. If omitted, CADDForge auto-detects common columns such as pose_file, molecule_file, ligand_file, path, or file.
• output_name: output filename for SDF filtering; defaults to filtered_molecules.sdf.

Behavior:

• If a path column is present, CADDForge uses those exact files first. For SDF files, it filters records inside the files and writes a filtered SDF. This is the preferred mode for sharded docking outputs where each shard has its own pose directory.
• If no path column is available and molecules points to a directory, files are retained when an identifier occurs in the file stem. The output artifact contains the retained file paths.
• If molecules points to an SDF-like file, records are retained when an identifier occurs in the record text. The output artifact is a filtered SDF file.
• If the downstream config field is typed as list[Path], CADDForge passes retained paths as a Python list. Gnina uses this through ligand_paths.
• Otherwise, CADDForge passes the artifact's primary path.

Example:

- id: retain_unidock_poses
  type: filter.molecules
  inputs:
    molecules: unidock_screen.poses
    ids: top_score_filter.filtered_table
  config:
    id_column: ligand

- id: gnina_rescore
  tool: docking/gnina
  config: gnina.yaml
  inputs:
    ligand_paths: retain_unidock_poses.molecules

Here, retain_unidock_poses.molecules exists because filter.molecules always emits an artifact named molecules.

To add a downstream workflow, append another stage. The input key must be a real config field on that tool, or an input port declared by the plugin:

  - id: downstream_analysis
    tool: my-section/my-downstream-tool
    config: downstream.yaml
    inputs:
      score_table_path: top_score_filter.filtered_table
    outputs:
      report: results_json

For Slurm, set the pipeline default backend and normal Slurm options:

defaults:
  backend: slurm
  registry_path: runs/job_registry.sqlite

Then pass optional CLI overrides as needed:

caddforge pipeline run --config pipeline.yaml \
  --slurm-partition short \
  --slurm-time 02:00:00 \
  --sbatch-option=--gres=gpu:1

Pipeline runs create a parent pipeline job under runs/YYYY-MM-DD/<pipeline-id>/. Child stage jobs, shards, reduced outputs, and artifact_manifest.json are stored under that parent run directory.

Pipeline Python API

The same pipeline engine can be used directly from Python. Build a PipelineSpec, then pass it to PipelineExecutor:

from pathlib import Path

from caddforge.pipeline import PipelineExecutor, PipelineSpec

spec = PipelineSpec.model_validate(
    {
        "version": 1,
        "name": "docking-screen",
        "defaults": {
            "backend": "local",
            "registry_path": Path("runs/job_registry.sqlite"),
            "job_name": "unidock-screen-test",
        },
        "stages": [
            {
                "id": "unidock_screen",
                "tool": "docking/unidock",
                "config": "unidock.yaml",
                "scatter": {
                    "by": "ligand_multisdf_path",
                    "chunk_size": 50000,
                },
                "outputs": {
                    "scores": "best_scores_csv",
                    "poses": "poses_dir",
                },
            },
            {
                "id": "top_score_filter",
                "type": "filter.tabular",
                "inputs": {
                    "table": "unidock_screen.scores",
                },
                "config": {
                    "sort_by": "best_score",
                    "ascending": True,
                    "top_n": 1000,
                    "output_name": "filtered_scores.csv",
                },
            },
            {
                "id": "retain_unidock_poses",
                "type": "filter.molecules",
                "inputs": {
                    "molecules": "unidock_screen.poses",
                    "ids": "top_score_filter.filtered_table",
                },
                "config": {
                    "id_column": "ligand",
                },
            },
            {
                "id": "gnina_rescore",
                "tool": "docking/gnina",
                "config": "gnina.yaml",
                "inputs": {
                    "ligand_paths": "retain_unidock_poses.molecules",
                },
                "outputs": {
                    "scores": "best_scores_csv",
                    "poses": "poses_dir",
                },
            },
        ],
    }
)

result = PipelineExecutor(spec).run()

print(result.pipeline_job.id)
print(result.pipeline_job.run_directory)
print(result.artifacts["top_score_filter.filtered_table"].primary_path())

If you already have a YAML/JSON pipeline file, use run_pipeline:

from caddforge.pipeline import run_pipeline

result = run_pipeline("pipeline.yaml")
print(result.pipeline_job.status.value)

---

Core Concepts

Dataset & MoleculeSet

• Dataset represents a reproducible, labeled collection of data (molecules + labels).
• MoleculeSet represents an immutable set of molecules.

Workflow

A Workflow orchestrates a protocol. It is backend-agnostic and delegates execution to a Backend.

Job

A Job is the canonical record of a workflow execution. It captures:

• identity (job_id), timestamps, and status
• run directory and metadata
• backend used

Backend

A Backend executes workflows. Core provides:

• LocalBackend (in-process execution)
• SlurmBackend (cluster submission via sbatch)

Result

A Result is a standardized, serializable outcome (status, metrics, paths, extra payloads).

Scratch synchronization

LogSync periodically mirrors live log files from node-local scratch without allowing transient copy failures to interrupt a workflow. ArtifactSyncQueue asynchronously publishes completed, immutable files through atomic replacement, retries unresolved copies on a fixed interval, and returns any files still pending after its final copy pass.

---

Configuration

All tools use Pydantic config models derived from BaseConfig. Configs can be loaded from YAML or JSON and are validated on load.

Example YAML (tool-specific fields omitted):

backend: local
run_directory: null
registry_path: runs/job_registry.sqlite

Base config also supports Slurm execution:

backend: slurm
slurm_partition: short
slurm_time: 01:00:00
slurm_account: my-account
sbatch_options:
  - --gres=gpu:1
slurm_nodes: 1
slurm_ntasks: 1
slurm_cpus_per_task: 1

---

CLI Reference

Open the documentation shipped by the optional caddforge-docs package:

python -m pip install caddforge-docs
caddforge docs
caddforge docs --host 0.0.0.0 --port 8001 --no-browser

This command appears only when the docs package is installed. It opens a browser and serves until Ctrl+C, using 127.0.0.1 and an automatically assigned port by default. --host supports hostnames, IPv4, and IPv6; --no-browser suppresses automatic opening. Documentation rendering tools are needed only to build the docs package.

List tools grouped by section:

caddforge list-tools

Show config template:

caddforge show-config <section> <tool>

Run a workflow:

caddforge run <section> <tool> --config path/to/config.yaml

Backend test:

caddforge backend-check --backend slurm --slurm-partition short

Status, logs, and job listing:

caddforge status <job-id>
caddforge logs <job-id>
caddforge logs <job-id> --follow
caddforge list-jobs --status running --status pending

Follow mode prints newly synchronized stdout.log content until the job reaches a terminal state. Transient log-file or job-registry read failures are reported and retried, which is useful when run directories reside on an intermittently available shared filesystem.

Export results manually (copies results/ back to the submission directory):

caddforge export <job-id>

---

Job Layout & Registry

Run directories follow a standardized layout:

runs/
  YYYY-MM-DD/
    <job-id>/
      config.yaml
      metadata.json
      stdout.log
      stderr.log
      results/
        results.json
        tool_specific/...

A SQLite registry tracks jobs and powers status/logs/list-jobs:

runs/job_registry.sqlite

By default, CADDForge leaves workflow outputs in the run directory. To also copy the results/ directory back to the submission directory automatically when a job finishes, opt in from the workflow config:

export_results: true

Manual export remains available through caddforge export <job-id> even when automatic export is disabled.

The registry uses short SQLite transactions with bounded retries for transient locking and I/O errors. For large Slurm fan-out runs, prefer placing registry_path on POSIX storage with working advisory locks. CIFS/SMB/NFS mounts with unreliable locking can leave the registry temporarily stale under high concurrency; per-job metadata.json and results/results.json remain the fallback source of truth. Do not use SQLite WAL mode for a registry shared across network-mounted hosts.

---

Backends

LocalBackend

• Executes workflows in-process.
• Captures stdout/stderr to log files.
• Saves results.json and updates metadata.

SlurmBackend

• Writes a payload and submits an sbatch script.
• Tracks status using sacct and squeue.
• Uses slurm_runner.py inside the job to execute the workflow callable.
• Requires the compute node to have access to a compatible Python environment; the login-node environment is usually not enough by itself.

Slurm options can be provided in config or via CLI flags:

• slurm_partition
• slurm_time
• slurm_account
• sbatch_options
• slurm_setup_commands
• slurm_env_pack_path / slurm_wheelhouse_path / slurm_requirements_path
• slurm_env_cache_enabled / slurm_env_cache_dir
• slurm_nodes
• slurm_ntasks
• slurm_cpus_per_task

Sharing the Python environment with Slurm

Slurm jobs execute on compute nodes, not in the interactive shell where you launched caddforge. In most cluster setups the compute node either cannot see the login-node virtualenv or cannot safely reuse it. For reproducible Slurm runs, provide one environment strategy in the workflow config.

Use slurm_env_pack_path when you want one prebuilt, relocatable environment. This is the preferred strategy for heavy scientific stacks because dependency resolution happens once, before submission. Use slurm_wheelhouse_path plus slurm_requirements_path when you want the Slurm job to create a fresh per-job virtualenv from a shared wheelhouse.

Option A: conda-pack / micromamba pack (single tarball)

1. Create an environment that contains:

   • caddforge-core
   • the tool plugin(s) you want to run
   • all scientific dependencies needed by those tools Install conda-managed scientific packages first, then install CADDForge packages with pip.

2. Use the core pack-env command to create the archive and managed SHA-256 sidecar. conda-pack is installed as a caddforge-core dependency:

   # Example environment creation with micromamba
   micromamba create -p /scratch/$USER/envs/caddforge -c conda-forge python=3.10
   micromamba install -p /scratch/$USER/envs/caddforge -c conda-forge openmm rdkit
   micromamba run -p /scratch/$USER/envs/caddforge python -m pip install caddforge-core <tool-packages>
   
   # Pack an explicit prefix
   caddforge pack-env --prefix /scratch/$USER/envs/caddforge \
     --output /shared/envs/caddforge.tar.gz
   
   # Or pack the active environment
   caddforge pack-env --output /shared/envs/caddforge-dev.tar.gz
   
   # Rebuild a fixed-name development pack
   caddforge pack-env --output /shared/envs/caddforge-dev.tar.gz --force
   
   # Pack an intentionally modified environment with missing Conda-managed files
   caddforge pack-env --output /shared/envs/caddforge-dev.tar.gz \
     --ignore-missing-files
   

   A named environment can be selected with --name. The command preserves conda-pack's progress bar as a single updating terminal line, streams normal output, reports quiet-period heartbeats with the growing archive size plus hashing progress, and writes <archive>.sha256 automatically.

3. Point CADDForge to the tarball:

   backend: slurm
   slurm_env_pack_path: /shared/envs/caddforge.tar.gz
   

   By default, CADDForge caches the unpacked environment in node-local scratch and reuses it for later jobs on that node. Set slurm_env_cache_enabled: false to retain job-private unpacking.

   The default per-user cache root is $SCRATCH_BASE/caddforge_environments/$USER, where SCRATCH_BASE is the same node-local root selected for Slurm job scratch. Sites with a different layout can provide an absolute compute-node path with slurm_env_cache_dir.

Option B: wheelhouse + per-job venv

1. Build a wheelhouse on a machine with internet access or your internal package index:

   # In a clean venv that has your deps installed:
   python -m pip freeze > requirements.txt
   python -m pip download -r requirements.txt -d /shared/wheelhouse
   

2. Provide the wheelhouse and requirements file:

   backend: slurm
   slurm_wheelhouse_path: /shared/wheelhouse
   slurm_requirements_path: /shared/requirements.txt
   

   The Slurm job will create a temporary venv, install from the wheelhouse, and execute the workflow with that Python.

Notes

• slurm_env_pack_path is mutually exclusive with slurm_wheelhouse_path / slurm_requirements_path.
• If you use slurm_setup_commands, do not set env-pack or wheelhouse options; those commands are then responsible for loading or creating the Python environment.
• CADDForge determines the environment-pack SHA-256 before submission. Cache entries include the archive name and digest. When <archive>.sha256 exists and is valid, submission reads that small sidecar instead of hashing the archive. Missing or invalid sidecars fall back to full hashing with a warning, preserving compatibility with manually created packs.
• Only the job installing a missing cache entry copies and verifies the archive; cache hits do not reread it. With slurm_env_cache_enabled: false, CADDForge skips sidecar lookup and submission hashing and restores the original job-private unpacking behavior.
• pack-env --force prepares and hashes the replacement before acquiring the publication lock, so submissions may continue using the previous valid pair while packing runs. Replacing an archive after a job was submitted can make an uncached queued job fail checksum verification.
• pack-env --ignore-missing-files passes the corresponding option to conda-pack. Use it only for intentionally modified environments: files recorded in Conda package metadata but missing from the source environment are omitted, so the resulting archive may be incomplete.
• The packaging lock is a separate hidden advisory file next to the archive. It never locks or modifies the source archive itself. Direct external archive replacement bypasses sidecar coordination and is therefore unsupported for managed packs.
• Cache locks are advisory files under the node-local cache root, never beside or on the source archive. Jobs and other compute nodes can continue reading the original pack concurrently.
• Installation uses a temporary directory and an atomic move. Active jobs retain a shared lock so opportunistic cleanup cannot remove their environment.
• Cache entries idle for more than 14 days are removed opportunistically. The retention period and 30-minute lock timeout are internal core policies rather than per-job configuration.
• Node-local caches may disappear during cluster cleanup or node replacement; CADDForge rebuilds a missing entry from the source pack.
• Ensure the shared paths are readable from compute nodes.

---

Plugin Architecture

Installed packages can also register optional top-level CLI commands in the caddforge.cli_commands entry-point group. The entry-point name becomes the command name and its target is a callable with a Typer-compatible signature and docstring. Core discovers these after registering built-in commands. Conflicting names, duplicate extensions, and broken imports or signatures are logged and skipped. This interface is separate from scientific tool discovery; command providers should keep imports lightweight and load expensive dependencies on invocation.

Tools register themselves as plugins via entry points:

[project.entry-points."caddforge.tools"]
ani-psa = "cadd_descriptors.tool:DescriptorsTool"

During development you can skip installation by setting:

export CADD_FORGE_DEV_PLUGINS=cadd_descriptors.tool:DescriptorsTool

The CLI will then discover tools on PYTHONPATH.

---

Development & Testing

Run tests:

pytest

Formatting/linting (if configured in your environment):

black .
ruff .

---

License

This project is intended for open-source use. Add your preferred license text here.

Continuous integration and releases

.github/workflows/release.yml builds a wheel and source archive, validates metadata, and tests the installed wheel on Linux with Python 3.10 and 3.12. The lightweight test selection is recorded in scripts/ci_tests.json; external scientific engines are validated separately. Core CI installs the wheel with its [test] extra, including RDKit for molecular hydrogenation coverage. RDKit remains optional for runtime installations. CLI help tests check both plain and styled output.

Pushing a matching v* release tag publishes the tested artifacts to PyPI. Branch pushes, pull requests and manual runs never publish. Configure a PyPI Trusted Publisher for the idorsia repository, release.yml workflow and pypi environment before pushing release tags. Core is published before the plugins; its runtime dependencies are resolved from public PyPI.