orchesjob-flow 0.1.0.dev0

Lightweight local flow runner built on top of orchesjob

orchesjob-flow

orchesjob-flow is a lightweight, daemonless local flow runner built on top of orchesjob.

It runs a small dependency graph of local commands, stores flow state in SQLite, and delegates each physical step execution to orchesjob. It is useful when you need more than a single orchesjob start, but do not want to introduce a full workflow engine, server, scheduler, or always-running daemon.

orchesjob-flow can be used from:

• a local terminal
• a server-side shell script
• cron or systemd timer
• CI/CD jobs
• SSH automation
• Apache Airflow / Amazon MWAA
• another in-house orchestrator

Airflow and MWAA are only examples of possible callers. They are not required.

Overview

orchesjob-flow is intentionally small:

• no web server
• no worker cluster
• no scheduler service
• no required daemon process
• no central control plane
• SQLite state on the host where the flow runs
• command-line interface only
• JSON output for automation
• table/watch output for humans

A flow is a YAML or JSON file containing named steps. Each step has a command and an optional when expression.

flow_id: daily-import
steps:
  - id: download
    command: ["python", "download.py"]

  - id: validate
    when: download
    command: ["python", "validate.py"]

  - id: import
    when: validate
    command: ["python", "import.py"]

  - id: notify_failed
    when: !download || !validate || !import
    command: ["python", "notify.py"]

The flow runner starts ready steps through orchesjob start, polls running steps through orchesjob status, and aborts running steps through orchesjob abort when needed.

Relationship with related tools

orchesjob

orchesjob is the single-job execution layer. It owns process execution, PID tracking, stdout/stderr files, exit status, run history, idempotent run keys, strict mode, rerun, abort, and result/status JSON.

orchesjob-flow does not replace orchesjob. It uses orchesjob for every physical step.

orchesjob-flow
  - decides which step can run next
  - evaluates step conditions
  - stores flow/step state in its own SQLite DB
  - calls orchesjob start/status/abort

orchesjob
  - starts the actual command
  - tracks the process
  - stores job execution state/history
  - owns stdout/stderr files
  - returns job status/result JSON

orchesjob-reserver

orchesjob-reserver is a reservation and dispatch layer for orchesjob. It stores a future execution intent and later dispatches it to orchesjob from a foreground dispatcher process.

orchesjob-flow is different: it is a local flow manager for multiple dependent steps. It does not require a dispatcher daemon. A flow can be run synchronously, detached into a background process, or advanced manually/recoverably.

orchesjob          = one physical job execution
orchesjob-reserver = reserve one job for later dispatch
orchesjob-flow     = run a small dependency graph of orchesjob-backed steps

Example call patterns

Local terminal:

user shell
  |
  v
orchesjob-flow start --run-key demo-001 flow.yaml
  |
  +-- orchesjob start/status for each step

Server-side automation:

cron / systemd timer / shell script
  |
  v
orchesjob-flow start -d --run-key nightly-20260507 flow.yaml
  |
  v
local background process continues until terminal flow state

Remote orchestration:

Airflow / MWAA / CI / SSH client / custom orchestrator
  |
  | SSH or remote command execution
  v
server or edge host
  |
  v
orchesjob-flow start -d --run-key daily-20260507 flow.yaml
  |
  +-- orchesjob start/status on the same host

The important point is that the caller only needs to start or inspect the flow. The detailed step execution happens locally on the host where orchesjob-flow runs.

Requirements

• Python >= 3.12
• orchesjob installed on the same host
• PyYAML

Installation

Recommended for CLI usage:

pipx install orchesjob-flow

For development:

git clone https://github.com/rmuraki/orchesjob-flow.git
cd orchesjob-flow
python -m venv .venv
. .venv/bin/activate
python -m pip install -e '.[dev]'

The installed command is:

orchesjob-flow --help

State directory and environment variables

orchesjob-flow stores its own SQLite database. By default, the database path is resolved in this order:

1. --db PATH
2. ${ORCHESJOB_FLOW_HOME}/orchesjob-flow.sqlite3
3. ${ORCHESJOB_HOME}/orchesjob-flow.sqlite3
4. ~/.local/share/orchesjob-flow/orchesjob-flow.sqlite3

Examples:

export ORCHESJOB_FLOW_HOME=/var/lib/orchesjob-flow
orchesjob-flow status --run-key demo

uses:

/var/lib/orchesjob-flow/orchesjob-flow.sqlite3

If ORCHESJOB_FLOW_HOME is not set but ORCHESJOB_HOME is set:

export ORCHESJOB_HOME=/var/lib/orchesjob
orchesjob-flow status --run-key demo

uses:

/var/lib/orchesjob/orchesjob-flow.sqlite3

orchesjob-flow and orchesjob have separate state. ORCHESJOB_HOME is passed through the environment to orchesjob naturally, so both tools can share a base directory if desired.

ORCHESJOB_BIN

Path to the orchesjob executable used by orchesjob-flow.

export ORCHESJOB_BIN=/usr/local/bin/orchesjob

Equivalent CLI option:

orchesjob-flow --orchesjob-bin /usr/local/bin/orchesjob status --run-key demo

Quick start

Create an example flow:

orchesjob-flow example > flow.yaml

Run it synchronously:

orchesjob-flow start --run-key demo-001 flow.yaml

Run it in the background:

orchesjob-flow start --run-key demo-001 -d flow.yaml

Watch it until it finishes:

orchesjob-flow watch --run-key demo-001

Show JSON status:

orchesjob-flow status --run-key demo-001

Show table status:

orchesjob-flow status --run-key demo-001 --format table

Cancel a flow:

orchesjob-flow cancel --run-key demo-001 --grace-seconds 10

Commands

Global options are accepted before the subcommand.

orchesjob-flow [GLOBAL_OPTIONS] COMMAND [COMMAND_OPTIONS]

Global options:

Option	Description
--version	Print the package version.
--db PATH	SQLite database path for flow state. Overrides environment-based default.
--orchesjob-bin PATH	orchesjob executable. Defaults to ORCHESJOB_BIN or orchesjob.
--poll-interval SECONDS	Sleep interval used by the flow loop. Default: 5.0.

start

Start or resume a flow run.

orchesjob-flow start --run-key KEY [-d|--detach] FLOW_FILE

Flags:

Option	Description
--run-key KEY	Flow run key. Required.
-d, --detach	Start a background child process and return immediately.
FLOW_FILE	YAML or JSON flow definition.

Without -d, start runs the flow loop in the foreground and returns after the flow reaches a terminal state.

orchesjob-flow start --run-key daily-20260507 examples/daily-import.yaml

With -d, start initializes/resumes the flow, spawns a detached background child process, and returns immediately.

orchesjob-flow start --run-key daily-20260507 -d examples/daily-import.yaml

Example detached output:

{
  "run_key": "daily-20260507",
  "flow_id": "daily-import",
  "status": "RUNNING",
  "started_at": 1778137200.123,
  "finished_at": null,
  "error": null,
  "steps": [
    {
      "id": "download",
      "run_key": "daily-20260507:download:1",
      "status": "PENDING",
      "attempt_no": 0,
      "max_retries": 0,
      "next_attempt_at": null,
      "started_at": null,
      "finished_at": null,
      "error": null,
      "orchesjob_status": null
    }
  ],
  "detached": true,
  "pid": 12345,
  "stdout_log": "/var/lib/orchesjob-flow/logs/orchesjob-flow-daily-20260507.stdout.log",
  "stderr_log": "/var/lib/orchesjob-flow/logs/orchesjob-flow-daily-20260507.stderr.log"
}

The background child runs the same flow loop that foreground start would run. It is not a persistent daemon. It exits when the flow becomes SUCCEEDED, FAILED, or CANCELLED.

status

Print the current flow state.

orchesjob-flow status --run-key KEY [--format json|table]

Flags:

Option	Description
--run-key KEY	Flow run key. Required.
--format json	Machine-readable JSON. Default.
--format table	Human-readable table.

JSON output is intended for automation, sensors, shell scripts, and CI jobs.

orchesjob-flow status --run-key daily-20260507

Example JSON output:

{
  "run_key": "daily-20260507",
  "flow_id": "daily-import",
  "status": "RUNNING",
  "started_at": 1778137200.123,
  "finished_at": null,
  "error": null,
  "steps": [
    {
      "id": "download",
      "run_key": "daily-20260507:download:1",
      "status": "SUCCEEDED",
      "attempt_no": 1,
      "max_retries": 0,
      "next_attempt_at": null,
      "started_at": 1778137202.456,
      "finished_at": 1778137262.789,
      "error": null,
      "orchesjob_status": {
        "run_key": "daily-20260507:download:1",
        "status": "SUCCEEDED",
        "exit_code": 0,
        "stdout_file": "/var/lib/orchesjob/logs/job.stdout",
        "stderr_file": "/var/lib/orchesjob/logs/job.stderr"
      }
    },
    {
      "id": "validate",
      "run_key": "daily-20260507:validate:1",
      "status": "RUNNING",
      "attempt_no": 1,
      "max_retries": 1,
      "next_attempt_at": null,
      "started_at": 1778137265.000,
      "finished_at": null,
      "error": null,
      "orchesjob_status": {
        "run_key": "daily-20260507:validate:1",
        "status": "RUNNING"
      }
    }
  ]
}

Table output is intended for operators.

orchesjob-flow status --run-key daily-20260507 --format table

Example table output:

Flow: daily-import  Run: daily-20260507
Status: > RUNNING  Steps: 1/3 done, 1 running, 0 failed
Started: 2026-05-07 10:00:00  Finished: -  Elapsed: 00:01:10

Step      Status        Attempt  Elapsed   orchesjob  Run key                         Error
--------  ------------  -------  --------  ---------  ------------------------------  -----
download  OK SUCCEEDED  1/0      00:01:00  SUCCEEDED  daily-20260507:download:1
validate  > RUNNING     1/1      00:00:10  RUNNING    daily-20260507:validate:1
import    . PENDING     0/0      -         -          daily-20260507:import:1

watch

Repeatedly display flow status until the flow reaches a terminal state.

orchesjob-flow watch --run-key KEY [--interval SECONDS] [--mode screen|append] [--advance]

Flags:

Option	Description
--run-key KEY	Flow run key. Required.
--interval SECONDS	Refresh interval. Default: 2.0.
--mode screen	Clear and redraw the same terminal screen. Default.
--mode append	Print a new table each interval. Useful for non-interactive shells, CI logs, and terminals that cannot use screen control.
--advance	Recovery/internal mode. Advance the flow once per refresh. Do not use with an already running detached loop for the same run key.

Default screen mode:

orchesjob-flow watch --run-key daily-20260507

Append mode:

orchesjob-flow watch --run-key daily-20260507 --mode append --interval 5

watch normally only reads state. It does not execute steps unless --advance is specified.

cancel

Cancel a flow run and abort currently running step jobs through orchesjob abort.

orchesjob-flow cancel --run-key KEY [--grace-seconds SECONDS]

Flags:

Option	Description
--run-key KEY	Flow run key. Required.
--grace-seconds SECONDS	Grace period passed to orchesjob abort.

Example:

orchesjob-flow cancel --run-key daily-20260507 --grace-seconds 10

Example output:

{
  "run_key": "daily-20260507",
  "flow_id": "daily-import",
  "status": "CANCELLED",
  "started_at": 1778137200.123,
  "finished_at": 1778137300.999,
  "error": null,
  "steps": [
    {
      "id": "download",
      "run_key": "daily-20260507:download:1",
      "status": "SUCCEEDED",
      "attempt_no": 1,
      "max_retries": 0,
      "next_attempt_at": null,
      "started_at": 1778137202.456,
      "finished_at": 1778137262.789,
      "error": null,
      "orchesjob_status": {"status": "SUCCEEDED"}
    },
    {
      "id": "validate",
      "run_key": "daily-20260507:validate:1",
      "status": "CANCELLED",
      "attempt_no": 1,
      "max_retries": 1,
      "next_attempt_at": null,
      "started_at": 1778137265.000,
      "finished_at": 1778137300.999,
      "error": "flow cancelled",
      "orchesjob_status": {"status": "RUNNING"}
    }
  ]
}

example

Print an example YAML flow to stdout.

orchesjob-flow example

Example:

orchesjob-flow example > flow.yaml

Hidden/internal command: tick

tick is intentionally hidden from normal help. It advances the flow engine once and exits.

orchesjob-flow tick --run-key KEY

It is kept for tests, recovery, and integrations that deliberately want to drive the flow loop externally. Normal users should prefer:

orchesjob-flow start -d --run-key KEY flow.yaml
orchesjob-flow watch --run-key KEY

Output model

Flow-level JSON fields

Field	Meaning
run_key	Flow run key supplied by the caller.
flow_id	Flow ID from the YAML/JSON definition.
status	Flow status. One of PENDING, RUNNING, SUCCEEDED, FAILED, CANCELLED.
started_at	Unix timestamp when the flow record was created.
finished_at	Unix timestamp when the flow reached a terminal state, or null.
error	Flow-level error string, or null.
steps	Ordered list of step state objects.

Step-level JSON fields

Field	Meaning
id	Step ID from the flow definition.
run_key	Physical orchesjob run key for the current attempt. Format: <flow-run-key>:<step-id>:<attempt-no>.
status	Step status. See below.
attempt_no	Number of attempts already started.
max_retries	Configured retry count.
next_attempt_at	Unix timestamp when a delayed retry becomes eligible, or null.
started_at	Unix timestamp of the current/last attempt start, or null.
finished_at	Unix timestamp when this step reached a terminal state, or null.
error	Step error summary, or null.
orchesjob_status	Last JSON object returned by orchesjob status, or null before the first poll.

orchesjob_status is stored as a pass-through payload. orchesjob-flow primarily uses its status field for step result decisions, while preserving the rest for operators and automation.

Status semantics

Flow statuses

Status	Meaning
PENDING	Flow record exists but the loop has not started yet.
RUNNING	At least one step is pending, delayed, or running.
SUCCEEDED	All steps reached terminal states and none failed. SKIPPED steps are acceptable.
FAILED	At least one step failed after retries were exhausted.
CANCELLED	Flow was cancelled.

Step statuses

Status	Meaning
PENDING	Step has not started yet.
DELAYED	Step is waiting for retry_delay_sec before the next attempt.
RUNNING	Step was started through orchesjob start.
SUCCEEDED	orchesjob status returned SUCCEEDED.
FAILED	Step failed and retries are exhausted. FAILED, LOST, CANCELLED, and ABORTED from orchesjob are failure outcomes.
SKIPPED	All referenced/waited steps are terminal, but the when condition is false.
CANCELLED	Flow was cancelled before the step completed.

Flow definition

A flow file can be YAML (.yaml, .yml) or JSON.

flow_id: branching-hello
steps:
  - id: A
    command: ["python", "examples/scripts/hello_step.py", "--flow-name", "branching-hello", "--step-id", "A"]

  - id: B
    when: A
    command: ["python", "examples/scripts/hello_step.py", "--flow-name", "branching-hello", "--step-id", "B"]

  - id: E
    when: !A
    command: ["python", "examples/scripts/hello_step.py", "--flow-name", "branching-hello", "--step-id", "E"]

  - id: D
    when: B || E
    command: ["python", "examples/scripts/hello_step.py", "--flow-name", "branching-hello", "--step-id", "D"]

Flow fields

Field	Required	Description
flow_id	yes	Logical flow name.
steps	yes	Non-empty list of step definitions.

Step fields

Field	Required	Description
id	yes	Unique step ID. Used in when expressions.
command	yes	Command array passed to orchesjob start -- ....
when	no	Boolean condition over step IDs. This is the primary execution condition.
depends_on	no	Additional wait-only dependency list. Results do not affect execution unless referenced by when.
retries	no	Number of retries after a failed attempt. Default: 0.
retry_delay_sec	no	Delay before retrying. Default: 0.
timeout_sec	no	Step runtime timeout. On timeout, orchesjob-flow attempts orchesjob abort and marks/retries the step.
strict	no	Whether to call orchesjob start --strict. Default: true.
start_timeout_sec	no	Passed to orchesjob start --start-timeout.

when expressions

when can be written as a small boolean expression over step IDs.

when: A
when: !A
when: A || B
when: A && B
when: (A && B) || C
when: (A && B) || !C

Expression semantics:

• A is true only when step A is SUCCEEDED.
• !A is true when step A has reached a terminal state and is not SUCCEEDED.
• FAILED, LOST, CANCELLED, ABORTED, and SKIPPED are all treated as non-success by !A.
• In expression mode, the condition model is currently success vs non-success. It does not distinguish failure from skipped.
• &&, ||, !, and parentheses are supported.
• Operator precedence is ! > && > ||.
• Use parentheses to make intent explicit.
• Steps referenced in when are automatically waited for before the expression is evaluated.

Example:

- id: D
  when: (B && F) || !C
  command: ["python", "finish.py"]

This waits until B, F, and C are terminal, then runs D if either:

B succeeded and F succeeded
or
C did not succeed

depends_on is wait-only

depends_on does not mean success dependency when when is explicitly provided. It only adds wait targets.

- id: D
  depends_on: ["cleanup"]
  when: C || F
  command: ["python", "finish.py"]

This means:

Wait until cleanup, C, and F are terminal.
Then run D if C or F succeeded.
cleanup's success/failure does not affect the condition.

If when is omitted, depends_on is treated as a normal success chain for compatibility.

- id: B
  depends_on: ["A"]
  command: ["python", "b.py"]

is equivalent to:

- id: B
  when: A
  command: ["python", "b.py"]

Recommended style for new flows is to use when for control flow and reserve depends_on for wait-only ordering requirements.

Dependency validation

Before starting a run, orchesjob-flow validates the effective dependency graph:

effective_dependencies = depends_on ∪ steps referenced by when

If the graph contains a cycle, the flow is rejected at start time.

Rejected example:

steps:
  - id: A
    when: C
    command: ["..."]

  - id: B
    when: A
    command: ["..."]

  - id: C
    when: B
    command: ["..."]

Example error:

{"error":"dependency cycle detected: A -> C -> B -> A"}

This prevents a flow from remaining forever in PENDING with no runnable step.

Retry behavior

Each retry uses a new physical orchesjob run key.

<flow-run-key>:<step-id>:1
<flow-run-key>:<step-id>:2
<flow-run-key>:<step-id>:3

Example:

- id: validate
  when: download
  retries: 2
  retry_delay_sec: 30
  command: ["python", "validate.py"]

Physical orchesjob run keys:

daily-20260507:validate:1
daily-20260507:validate:2
daily-20260507:validate:3

This avoids reusing a completed strict orchesjob run key for retry attempts.

Timeout behavior

If a step has timeout_sec, orchesjob-flow checks elapsed time while polling the running step.

- id: import
  when: validate
  timeout_sec: 3600
  command: ["python", "import.py"]

When the timeout is exceeded:

1. orchesjob-flow attempts orchesjob abort for that step run key.
2. The step is treated as failed for that attempt.
3. If retries remain, the step moves to DELAYED or PENDING for the next attempt.
4. If retries are exhausted, the step becomes FAILED.

Exit codes

Command	Success exit code	Failure exit code
start foreground	0 if final flow status is SUCCEEDED; non-zero if FAILED or CANCELLED.
start -d	0 if the background child was launched. Later flow failure is observed through status or watch.
status	0 if the flow record exists and status can be printed.
watch	0 if final flow status is SUCCEEDED; non-zero if FAILED or CANCELLED.
cancel	0 if cancellation command completes.
validation / CLI error	2 with JSON error on stderr.
interrupted	130.

Examples included in this repository

File	Purpose
examples/daily-import.yaml	Simple sequential import flow with failure notification.
examples/branching-hello-flow.yaml	Branching flow: A-B-C-D and A-E-F-D.
examples/branching-hello-failure-flow.yaml	A intentionally fails; failure branch E-F-D runs.
examples/branching-hello-parens-flow.yaml	Demonstrates parentheses in when, such as `(B && F)
examples/scripts/hello_step.py	Emits Hello <flow-name> every 5 seconds for about 1 minute.
examples/scripts/hello_step_maybe_fail.py	Same as above, with optional intentional failure.

Operational patterns

Fully local

orchesjob-flow start --run-key local-demo examples/branching-hello-flow.yaml

Detached local/background flow

orchesjob-flow start --run-key local-demo -d examples/branching-hello-flow.yaml
orchesjob-flow watch --run-key local-demo

Server-side script

#!/usr/bin/env bash
set -euo pipefail
export ORCHESJOB_HOME=/var/lib/orchesjob
orchesjob-flow start --run-key "nightly-$(date +%F)" -d /opt/flows/nightly.yaml

Remote command from another machine

ssh app-server 'ORCHESJOB_HOME=/var/lib/orchesjob orchesjob-flow start --run-key daily-20260507 -d /opt/flows/daily.yaml'
ssh app-server 'ORCHESJOB_HOME=/var/lib/orchesjob orchesjob-flow status --run-key daily-20260507 --format table'

Airflow/MWAA-style usage

Airflow/MWAA can be one caller, but it is not special to the tool.

Typical pattern:

Task 1: SSHOperator or equivalent
  orchesjob-flow start --run-key <dag-run-key> -d /opt/flows/flow.yaml

Task 2: Sensor or polling task
  orchesjob-flow status --run-key <dag-run-key>

Task 3: Optional log/result collection
  use orchesjob run keys from status output

What this tool is not

orchesjob-flow is not:

• a replacement for Airflow
• a replacement for MWAA
• a replacement for Temporal
• a distributed workflow engine
• a cron scheduler
• a web UI
• a long-running central service
• a cluster manager

It is deliberately a small local flow layer for cases where a handful of dependent steps should run on one host with durable local state.

Development

Install development dependencies:

python -m pip install -e '.[dev]'

Run tests:

pytest

Run lint/type/build checks:

ruff check .
mypy
python -m build

License

MIT