YAML Workflow Framework

YAML workflow engine — parses, validates, and executes step-based workflows with pluggable Action / Evaluator / Predicate / Iterable methods.

Overview

vcti-ywf is the workflow engine behind VCollab Insights. A workflow is a directory of YAML files — about.yaml, variables/main.yaml, steps/main.yaml — that describe an automated process declaratively. The engine parses those files (with support for !var, !include, and !module tags), resolves variables, evaluates control-flow blocks (if-else, for-each, try-except), and dispatches each step to a registered method plugin.

Method plugins come in four typed shapes — Action (side-effecting), Evaluator (produces a value), Predicate (boolean test), and Iterable (yields keys for for-each). A built-in stdlib package ships common methods; downstream consumers add domain-specific methods (CAE data extraction, plotting, report generation) by registering their own classes.

Workflow YAML is evaluated against a sandboxed Context by default — - evaluator: evaluate expressions go through simpleeval, and {{ var }} template interpolation goes through Jinja2's SandboxedEnvironment. Engine-internal context variables (__runtime__ and friends) are filtered from author-facing scope. Hosts that run fully-trusted YAML can opt into raw eval or the full Jinja2 environment by constructing the Context themselves and passing it to Runner.run().

Installation

pip install vcti-ywf>=3.0.1

In requirements.txt

vcti-ywf>=3.0.1

In pyproject.toml dependencies

dependencies = [
    "vcti-ywf>=3.0.1",
]

Quick Start

Run a workflow from Python:

from pathlib import Path

from vcti.ywf import Runner, Workflow

workflow = Workflow(Path("path/to/my-workflow"))
workflow.read_all()

result = Runner(workflow).run({"input_file": "data/model.h5"})

print(result.return_value)   # top-level block's value
print(result.outputs)        # declared outputs read from final context

For monitoring or debugging, build an ObservableContext, attach observers, and pass it to run():

from vcti.ywf import ObservableContext, Runner
from vcti.ywf.execution.observers.history_observer import HistoryObserver

context = ObservableContext()
context.add_observer(HistoryObserver())

Runner(workflow).run({"input_file": "data/model.h5"}, context)

A minimal workflow directory looks like:

my-workflow/
├── about.yaml              # title, description, tags, images
├── variables/
│   └── main.yaml           # inputs + outputs declarations
└── steps/
    └── main.yaml           # execution steps

variables/main.yaml declares the workflow's external contract — inputs (supplied by the caller) and outputs (produced by the run):

inputs:
  - name: input_file
    type: input_file_path
    description: Path to the simulation results
    default: data/sample.h5

outputs:
  - name: mode_count
    type: int
    default: 0
    description: Number of modes extracted.

steps/main.yaml lists actions, evaluators, predicates, and iterables — each is a YAML mapping keyed by its method name. The data-scope actions illustrate the typical pattern:

- action: define_data_scope
  name: simulation
  type: paths-group

- action: add_path_source
  path: !var input_file
  format: hdf5-file
  label: results

- action: load_data_scope

- if:
    predicate: is_data_scope_valid
  then:
    - action: log_message
      message: "Data scope ready."
  else:
    - action: log_message
      log_level: error
      message: "Data scope failed to load."

Documentation

User guides — start here:

• docs/tutorial.md — walk through the greeting workflow end-to-end with the Python API, then expand to control-flow blocks and data-scope actions.
• docs/yaml-reference.md — exhaustive field reference for about.yaml, variables/main.yaml, steps/main.yaml, and every step type (action / evaluator / predicate / iterable / if-else / for-each / try-except).
• docs/yaml-tags.md — !var, !include, and !module semantics with scoping rules.
• docs/plugins.md — author guide for custom workflow methods.
• docs/api.md — autogenerated API reference.

Developer reference — engine concepts subpackage by subpackage:

• docs/developer/README.md — index for the developer documentation.
• docs/developer/core/workflow-structure.md — how a workflow is organised on disk.
• docs/developer/core/variables.md — inputs: / outputs: sections, host-facing helpers.
• docs/developer/core/runtime.md — Runtime, Runner, RunResult, caller-supplied context.
• docs/developer/core/execution-context.md — the sandboxed Context, trust posture, custom Jinja2 filters.
• docs/developer/core/methods.md — method registry, dispatchers, plugin authoring.
• docs/developer/core/flow.md — Block variants and dispatch.
• docs/developer/core/dsl.md — Value / Property, ContextBase, executables.
• docs/developer/core/stdlib.md — built-in actions, evaluators, predicates, iterables.

Internals — for engine maintainers and tool builders:

• docs/developer/internals/overview.md — source-location tracking, observation, and control as three interlocking pieces.
• docs/developer/internals/dsl.md — MetadataMixin, Source / StepMetadata, the _meta field.
• docs/developer/internals/yaml-loader.md — the three loader classes, subdir_dir / current_file invariants, !var / !include / !module implementation.
• docs/developer/internals/execution-observing.md — ObservableContext, the event hierarchy, built-in observers, the ExecutionController.
• docs/developer/internals/debug-command.md — how to build a step-by-step debugger on top of the controller + observer pair.

Dependencies

Third-party:

• pydantic (>=2.0) — model validation
• PyYAML (>=6.0) and pyyaml-include (>=2.0) — YAML parsing with !include
• Jinja2 (>=3.0) — {{ var }} interpolation in Property[str] fields (sandboxed via SandboxedEnvironment in the default Context)
• simpleeval (>=1.0) — sandboxed expression evaluator for - evaluator: evaluate
• NumPy (>=1.24) — array handling via vcti-array-tree
• pandas (>=2.0) — CSV reading in the read_file evaluator and DataFrame iteration in list_items

VCollab platform packages:

• vcti-array-tree — hierarchical array data
• vcti-config — environment-variable decoding (get_env evaluator)
• vcti-data-scope — DataScope / PathsGroup types used by the data-scope actions
• vcti-enum — value-generated enums + Pydantic helpers
• vcti-error — error codes
• vcti-fileloader — pluggable loader registry
• vcti-logging — logger
• vcti-lookup — Rule-based attribute filtering
• vcti-path — safe path & filename validation
• vcti-path-format, vcti-path-format-descriptors — file-format identification
• vcti-path-tree — directory tree scanning

Optional extras:

• vcti-ywf[hdf5] — pulls in vcti-fileloader-hdf5, which Runner auto-registers if installed.