kestrel-feature-workflows

First-class agent workflow orchestration for Kestrel Sovereign.

The package also installs a host-scoped WorkflowsHostFeature that owns the durable, engine-neutral operator run plane. It registers the public SDK RunService at host scope as workflows.runs version 1.1.0, persists accepted launches and their recovery outbox in the host database, and exposes a fenced producer lifecycle for installed execution features. The run plane does not add HTTP routes or engine-specific fields.

Absolute artifact links fail closed unless their exact HTTPS origins are listed in the package-owned KESTREL_WORKFLOWS_ARTIFACT_ORIGINS environment variable as a JSON array, for example ["https://artifacts.example.com","https://cdn.example.com:8443"]. Entries must be exact HTTPS origins: paths, queries, fragments, credentials, and non-HTTPS URLs are rejected. The value is limited to 16 KiB, 100 origins, and 2,048 characters per origin. Canonical /authorized/artifacts/<artifact_id> links remain same-origin and need no allowlist entry.

Alternate hosts may instead provide the existing workflows.artifact_origins context mapping. If both sources are present, their normalized origin sets must be identical or host-feature startup fails; otherwise the environment source is used when set. Malformed values fail startup without logging their contents.

Workflows is orchestration on top of the existing Signal Dispatcher. Stages dispatch signals (ACTION / ARTIFACT / COGNITION); durability, dedupe, locks, causation tracking, retention, and redaction all come from the dispatcher. Workflows adds what the dispatcher lacks: multi-stage sequencing and fan-out, per-stage gates (including an adversarial red_team_clear gate), reverse-order saga compensation, a cancellation barrier, DID-signed versioned workflow definitions, and constitutional-boundary enforcement.

Execution is lease-owned: runnable runs and open stage/compensation attempts are atomically claimed by an executor and heartbeated. Recovery reuses the persisted attempt idempotency key and dispatch linkage; it advances confirmed effects, never reruns completed stages, and fails safely when an external effect cannot be determined. This provides exactly-once workflow transitions and at-most-once effects where downstream idempotency supports it, not a blanket exactly-once guarantee for arbitrary external systems.

Declared runtime triggers

WorkflowSpec.triggers is honored at runtime for manual, cron, and signal_source declarations. Manual behavior is unchanged. A non-manual definition is rejected before persistence when its static trigger parameters do not satisfy params_schema, its signal source is not registered, or the required released Core capability is absent.

Cron declarations are registered only through SchedulerFeature's public schedule_add / schedule_list / schedule_remove contract. They run in the signed IANA timezone_name (UTC when omitted for backward compatibility) with misfire_policy="skip"; each callback must carry Scheduler's public, non-forgeable execution identity. The scheduled occurrence time, not callback time, is part of the deterministic workflow launch identity. Signal-source declarations use SignalDispatcher's public agent-scoped durable consumer registration and claim/lease/ACK/NACK lifecycle. Provider source_event_id deduplication and the normalized durable event ID make redelivery converge on one workflow run. A versioned agent/source boundary is captured before the definition and desired registration commit, so retained history does not launch retroactive runs while post-commit events survive registration/restart windows.

Timezone names must use the canonical case-sensitive IANA spelling (for example, America/Chicago). New definitions are validated against the installed tzdata release (version 2026.3 or newer). Scheduler execution and reconciliation use Core's host timezone resolver, including system zoneinfo. Deployments should keep installed timezone data consistent across executors. During a rolling Workflows upgrade, all executors reconciling the same scope must run version 0.7.0 or newer before admitting a definition with an explicit non-UTC timezone_name. A pre-0.7.0 reconciler drops that signed field while rebuilding desired state, hits trigger registration identity conflict, and aborts reconciliation for the entire owner/dispatcher scope. The existing schedule remains intact, but retirement, drift repair, restart adoption, and health updates do not converge until every reconciler is upgraded.

The package persists an owner-and-live-dispatcher-scoped desired-state ledger for every public registration. Definition writes, startup, soft re-enable, revoke, feature disable, and feature removal reconcile that ledger. Retirement intent is committed before the external resource is removed or deactivated; publication identity is committed before SignalDispatcher registration and Scheduler rows are adopted from their exact public arguments/idempotency key after an ambiguous add result. This closes the supported cross-store crash windows without reading Scheduler or SignalDispatcher private tables and without claiming a transaction across their stores.

Signed execution context and action outputs

A stage may explicitly set include_execution_context=true to receive a reserved workflow_execution object containing its stable run ID, stage name, attempt number, and stage idempotency key. Existing stages receive no new payload fields. A signed output_projection maps bounded RFC-6901 pointers in an action result to run-parameter names. It is supported on gates that settle in the dispatch attempt; pending gates (await_signal, consent_collect, and council_approve) reject it at definition time. Projected JSON is persisted with the public effect receipt, and the successful stage transition publishes those parameters atomically, so subsequent stages and await_signal context materialization see the same values after normal execution or crash recovery. Reserved workflow identity parameters in legacy stage, gate, or declared trigger params are ignored at dispatch; new manual launch params, merge targets, and projection targets reject them. This preserves old signed definitions without permitting them to forge execution identity. The additive workflow_stage_links.effect_output_params_json column is installed idempotently at initialization, with {} as the rolling-upgrade default. Before admitting a definition that uses include_execution_context or output_projection, every definition, run, and recovery executor in that owner scope must run Workflows 0.7.0 or newer. A pre-0.7.0 recovery executor drops either signed field while verifying the pinned definition, claims the expired run, and terminally fails it as recovery_error_no_safe_resume without compensating already-completed stages. Cron timezone names are checked against the installed tzdata catalog when a new definition is admitted. Persisted signed definitions remain loadable if a later catalog removes an alias; reconciliation then marks only that scheduler registration unhealthy and removes its stale schedule without failing live workflow runs.

Signal trigger delivery is drained in bounded pages by the feature's existing supervised lease-recovery cadence; declared triggers add no second polling task. SchedulerFeature remains the cron wake owner and SignalDispatcher remains the normalized-signal/durable-delivery owner. The default cadence can add up to roughly ten seconds of signal-trigger launch latency. A trigger run persists its registration, delivery/event or schedule identity, deterministic idempotency key, and causation evidence before the transport is ACKed or the Scheduler callback succeeds. A crash after that handoff is recovered through the ordinary workflow run lease.

Signal events do not implicitly become workflow input. Only the trigger's signed, static params are passed as run.params; triggering event data is not projected into run.params. This includes its payload, caller identity, and event identity. Event and delivery identifiers remain available as run causation and inspection evidence. A future payload mapping must be an explicit signed, schema-validated declaration; this release never merges untrusted event data into stage parameters.

workflow_list_definitions includes registration state and workflow_inspect_definition returns trigger_health, lifecycle state, last error, and the last delivered event/run identity. Unhealthy registration or cleanup is returned as a partial tool result rather than hidden. Owner DID, live agent/dispatcher DID, definition/version/hash, and public resource identity are checked again at every callback and delivery; a scope mismatch is never reassigned to the current user.

workflow_inspect_retention exposes only the live owner's durable retention debt. A terminal run missing its exact owner/name/version definition is retained until that signed definition is restored; the next sweep clears the receipt set-wise and applies the restored retention policy. This avoids guessing a retention period or deleting another tenant's evidence.

See the trigger architecture and migration notes for identities, crash windows, and operational behavior.

Workflows persists the public durable-admission disposition and terminal SignalResult for each await effect. Core 0.53.3 has no public historical effect-status lookup, so a process loss after admission but before that result is persisted remains durably COMPENSATING unless the host supplies workflow_effect_status_resolver backed by authoritative public effect evidence. Workflows never queries Core's private signal_log table to guess.

Durable await_signal gates

An effect stage may use an await_signal gate to dispatch once, then wait for an authenticated Core signal without retaining an asyncio task. The gate requires a registered source, absolute timeout, matcher_version: 1, bounded all/any matcher (eq, in, exists over JSON Pointers), and optional sanitized-payload projection. Only the closed gate.passed branch is permitted: its true target is the signal path and its false target is the timeout path. Workflows persists the consumer, tenant/owner scope, frozen context operands, deadline idempotency key, and terminal evidence. Immediately before dispatching the external stage effect, Workflows captures Core 0.53.3's public DurableSourceBoundary for the exact agent/source scope and commits its complete versioned record in the wait row. The row remains in boundary_captured setup state until effect admission; a restart reuses it and never captures a later replacement. A claimed delivery and the scheduler deadline race through one store transaction, so a late or duplicate delivery can be audited but cannot replay the effect or graph transition.

Core 0.53.3 provides the durable boundary and delivery lifecycle, but its DurableSignalEvent does not persist an immutable, authenticated receipt of the sanitizer and effective trust policy that applied at ingress. A sanitizer registered today is not proof about a retained historical event. Therefore Workflows 0.6.0 fails closed before the await effect dispatches unless the host provides workflow_await_signal_event_trust_verifier, backed by a newer public Core per-event attestation contract. Every delivered receipt is checked against the event ID, dispatcher agent DID, source, and source sequence. The required host verifier authenticates the receipt and validates its policy epoch against the host's ingress-policy history; Workflows can only require a non-empty epoch because Core 0.53.3 exposes no public policy-epoch provider to pin independently. Workflows does not infer this proof from current registration and does not inspect private Core tables. ACTION delivery remains ineligible because it bypasses sanitization.

The host must also provide workflow_await_signal_privacy_transition_lock through a public integration surface. Core 0.53.3 keeps its corresponding agent lock private, so Workflows does not reach into it; without the injected public lock the gate fails closed before effect dispatch.

Core 0.53.3 serializes boundary capture, signal commit, and registration backfill through one durable agent/source sequence domain on SQLite and PostgreSQL. An event is eligible only when its source_sequence is strictly greater than the persisted boundary sequence. committed_at, activation time, delivery identity, and registration time are never source-boundary eligibility fallbacks. Deadline eligibility is separate: a matching event wins only when Core's linearized committed_at is strictly before the persisted deadline; a commit exactly at the deadline belongs to timeout. Polling time never changes that result. Timeout reconciliation gives publicly visible, nonterminal pre-deadline deliveries a fixed 120-second grace after the persisted deadline—two complete Core delivery-lease periods—to settle or redrive. The grace is absolute and survives restart; reconciliation never extends it. After it expires, timeout may win even when a source remains disabled/unregistered, another executor still holds the lease, or Core's public 1000-row inspection page is full. deadline_won evidence records every visible unsettled pre-deadline delivery ID and whether that page hit the cap. Malformed public rows without usable delivery IDs and failed initial drains are represented by redacted synthetic unsettled identifiers; they receive the same bounded grace and cannot bypass timeout liveness. Core's public durable-consumer registration has no boundary field, so backfill may materialize retained pre-boundary deliveries. Workflows validates every public delivery against its persisted boundary, terminally NACKs a claimed pre-boundary lease, and continues draining so a later post-boundary event is not stranded. If Core claims the post-boundary row first, consumer cleanup may terminalize the older row without creating Workflows evidence. A legacy row without the complete boundary stays inspectable and cleanable after the additive migration but fails closed for publication and resolution, including on PostgreSQL.

Workflows uses a durable publication lease plus an orphan_guard tombstone around the separate Core and Scheduler calls. Cancellation and force-abort serialize with local publishers, remove exact returned resources after ownership loss, and retain an unpurgeable retry intent after an interrupted call whose eventual commit cannot be disproved. This is deliberately not described as a cross-store atomic transaction.

Exact session selectors isolate runs by default; agent-wide broadcast is a signed opt-in for INTERNAL events within the same user-agent tenant. Public, external, and A2A-relayed events cannot win an agent-scoped wait; A2A authority remains a separate surface. Recovery reads bounded owner/tenant/status-scoped SQL pages through durable, wrapping cursors, quarantines malformed rows with redacted structural audit evidence, and takes at most one public Scheduler snapshot per pass. Environmental registry reloads NACK for unbounded redelivery; immutable scope mismatches ACK. Volatile, anonymized, and deidentified privacy transitions are checked under Core's transition lock and atomically fence open waits. The deadline winner takes the same lock and rechecks privacy before its row-locked CAS. Timeout CAS eligibility and retention cleanup use database time; signal arrival eligibility uses Core committed_at. Terminal recovery converges the Core consumer and every related live Scheduler task while preserving fired terminal one-shots as Scheduler audit history; purge waits for both cleanup facts. A retention sweep supplied with the live dispatcher DID never performs cleanup for an older/foreign dispatcher tenant: after retention expires it persists a redacted stranded-resource receipt, visible through list_stranded_await_signal_retention, and then purges the run. Omitting live dispatcher context remains fail closed and lets outstanding cleanup pin the run. Timeout wins compensate only the completed await effect before the false branch proceeds; earlier PASS or irreversible stages remain part of the live workflow. Shutdown drains boundedly and retains unknown-effect evidence and leases for recovery when a handler resists cancellation.

Delivery progress is currently periodic Workflows reconciliation, not a Core event-driven callback. The feature reconciles once after initialization and then every min(lease_seconds / 3, 10 seconds) (with a 50 ms floor), so the default 30-second lease gives up to roughly 10 seconds of wake latency plus processing time. Scheduler one-shots provide lower-latency timeout wakeups, but each reconciliation/restore pass also scans a fair bounded database-clock lane and resolves overdue waits through the same winner CAS, so timeout liveness does not depend on Scheduler callback delivery. Availability requires the Workflows recovery worker, its database, and Core durable delivery APIs; Scheduler list/remove availability is still needed to converge live task cleanup while preserving terminal Scheduler rows as audit history. A future Core event-driven caller may reduce signal latency but is not part of this release.

Installation

uv pip install kestrel-feature-workflows

The package registers WorkflowsFeature through kestrel_sovereign.features and WorkflowsHostFeature through kestrel_sovereign.host_features.

Operator Runs API and console

Workflows 0.6.0 exposes its durable, engine-neutral run plane at /api/workflows/runs. Authenticated operators can page and filter run records, inspect stages, numbered attempts, typed external job correlations, and artifact metadata, apply pause/resume/cancel/retry controls with idempotency and sequence preconditions, and follow one authorized run through a polling-backed SSE stream. Cross-tenant and absent identifiers share the same not-found response, and artifact dereference returns metadata only.

WorkflowsHostFeature also contributes the capability-gated Runs panel to the Console from package-owned JavaScript and CSS. The panel uses authenticated streaming with polling fallback, renders generic SDK fields only, and offers an optional correlation handoff to an installed Observability panel.

An injected workflows.operator_context_resolver remains authoritative for Castle and other multi-tenant deployments. Otherwise Workflows resolves the current host-scoped claws.catalog>=1.3 service for each request and asks its public operator-context resolver to authorize OAuth and other authenticated solo operators. A present Claws service is authoritative: policy denial or an invalid resolver fails closed and cannot fall through to broader local rights. Generic Workflows hosts without a compatible Claws service retain the existing zero-config fallback, which grants access only to the Sovereign API-key caller and requires an injected tenant resolver. A present 1.2 catalog without the 1.3 operator policy is not considered absent and fails closed. The selected resolver is responsible for user, tenant, boundary, capability, action, and expiry policy.

The HTTP and Console surfaces intentionally do not launch runs. Launch remains the feature-owned RunService.launch_run boundary used by installed execution features, where source attribution, target boundaries, capabilities, tenancy, and idempotency are authorized before acceptance. No Talon-, Eye-, Flight-, repository-, command-, or filesystem-specific fields are part of this API.

Development

uv sync --extra test
uv run --extra test pytest -q
npm test

Design

See docs/architecture/WORKFLOWS_FEATURE_DESIGN.md (v4.1) and WORKFLOWS_DEVELOPER_GUIDE.md in the kestrel-sovereign repository for the full design and the stage-to-signal reduction.