palmengine 0.9.7

Palm Engine — lightweight orchestration for multi-step transactional workflows

Palm Engine 🌴

Palm is a lightweight, Python-first orchestration engine built on a clean Behavior Tree foundation. It coordinates interactive wizards, data pipelines, and—over time—compute-heavy workloads with explicit contracts, durable state, and human-first tooling.

Current release line: 0.9.7 · See CHANGELOG.md · MIGRATION-0.6.md · SCOPE.md for roadmap

---

Installation

Palm is published on PyPI as palmengine. After install, you import palm and run the palm CLI — same names as in source development.

What	Name
PyPI package	palmengine
pip install	pip install palmengine[cli]
Python import	import palm
CLI command	palm

# End users — CLI + REPL
pip install palmengine[cli]
palm version --full
palm doctor

# Library only (no Rich / REPL)
pip install palmengine

# From source (contributors)
git clone https://github.com/JGabrielGruber/palmengine.git && cd palmengine
uv sync --group dev --extra cli
uv pip install -e ".[cli]"

Optional extras: [cli], [test], [dev], [all], [postgres], [mongodb].

---

Vision

Palm aims to be simple at the core and powerful at the edges:

• Human-first — interactive wizards, Rich CLI feedback, backtracking, resume after interruption
• Truth-seeking — pluggable state, persistent process instances, transactional commits
• Extensible — patterns, providers, and storages register at the edge; core stays pure
• Ambitious but honest — from onboarding wizards to multi-flow data pipelines and planned GPU kernel nodes

Behavior Trees are the control-flow foundation. Steps are nodes. Cross-cutting concerns (auth, guards, observability) belong in runtimes and optional BT guard nodes—not buried in step definitions.

---

What works today (0.9.7)

Area	Capabilities
Core	Behavior tree, orchestration (apply_result authority), context, storage, resource, event, auth, TransformEngine
State	DictStateSchema (incl. length/item constraints), scoped state, schema-aware snapshots (__palm:meta), observability events
Transforms	22 built-in rules — field shaping, JSONPath, dates, conditionals, and serialization (JSON/CSV/YAML/TOML/XML)
Patterns	Wizard (layered validation, collection steps, step_kind: transform, summary/commit); parallel branches; DAG and ETL stubs
Executions	ExecutionPlan / ProcessPlan, DefinitionExecutor, prepare/submit batch API
Persistence	Production filesystem backend (path-safe), StorageFactory, InstanceManager, durable resume across restarts
State snapshots	Optional StateSnapshotHook — bounded blackboard history for audit/debug (off by default)
Runtimes	EmbeddedRuntime, DaemonRuntime, ServerRuntime (HTTP), CLI + REPL
Middleware	JobHook, AuthMiddleware, drive observability, instance persistence, state snapshots
DX	Examples (transform-example, transform-formats, todo-builder, parallel-demo), palm doctor transform catalog, just quality recipes

flowchart LR
    User[Developer / operator] --> CLI[CLI / REPL]
    CLI --> ER[EmbeddedRuntime]
    ER --> CM[common]
    CM --> PAT[patterns]
    CM --> INST[instances]
    PAT --> BT[Behavior Tree]
    INST --> STO[storage]

---

Quick start

pip install palmengine[cli]

palm version --full      # version + registered plugins
palm doctor              # health, definitions, instances
palm repl                # interactive shell (default: `palm`)
palm flow start onboard          # recommended — works for all patterns
# shortcut: palm start onboard

From source: uv sync --group dev --extra cli && uv pip install -e ".[cli]" then the same palm commands. Demo script: uv run python examples/full_demo.py.

Try the new examples:

palm flow start schema-onboard   # layered state schemas + scopes
palm flow start todo-builder     # dynamic todo list (collection step)
palm flow start parallel-demo    # parallel wizard branches
palm flow start transform-example  # wizard transform steps
palm flow start transform-shaping  # pipeline calculate / lookup / conditional
palm flow start transform-formats  # json_load → csv_dump ETL-style pipeline

Transforms (0.9.7)

Declarative data shaping via registered rules — usable in pipelines, wizard steps (step_kind: transform), or TransformLeaf nodes.

from palm.common.transforms import TransformExecutor, autoload

autoload()
executor = TransformExecutor()
result = executor.apply(
    "string_format",
    "ada",
    template="Hello, {value}!",
    case="title",
)
# → "Hello, Ada!"

# Wizard step (in flow options.steps)
step_kind: transform
source_key: name
target_key: greeting
rule: string_format
options:
  template: "Hello, {value}!"
  case: title

Run palm doctor for the full rule catalog with descriptions. Extend with register_transform("my_rule", MyRule) at bootstrap.

CLI persistence: the CLI is a thin client of PalmApp. By default it uses in-memory storage (fast, non-durable). Set durable storage via flags or environment:

# Recommended for local work — persists instances under ./data/
export PALM_STORAGE_BACKEND=filesystem
export PALM_DATA_DIR=./data

# Or per invocation:
palm --storage-backend filesystem --data-dir ./data wizard start onboard

palm doctor and REPL startup show whether state will survive restarts. Instance commands (list, status, snapshots, resume, prune) all resolve through the same PalmApp.instance_manager — short ids from instance list work with prefix matching.

Global CLI flags (override env only when explicitly passed):

Flag	Env	Purpose
-b / --storage-backend	PALM_STORAGE_BACKEND	Storage backend (memory, filesystem, …)
-d / --data-dir	PALM_DATA_DIR	Data directory for durable backends
--config	—	Optional .env-style config file
-S / --enable-state-snapshot	PALM_ENABLE_STATE_SNAPSHOT	Capture state snapshot history
--max-loaded-instances	PALM_MAX_LOADED_INSTANCES	InstanceManager LRU size
--max-concurrent-active	PALM_MAX_CONCURRENT_ACTIVE	Active instance cap
--scheduler	PALM_DEFAULT_SCHEDULER	inline or queued
--format	—	table (default) or json for scripting

Settings precedence: PALM_* environment → --config file → CLI flags.

palm instance list                          # active (non-terminal) instances
palm instance list --all --format json      # all instances, JSON for scripts
palm instance list --status WAITING_FOR_INPUT --flow quick
palm instance prune --dry-run               # preview terminal instance cleanup
palm --format json instance status <id>     # machine-readable status

The REPL uses smart tab-completion for commands, flow/process names, and instance ids (active by default; --all includes terminal instances).

---

Persistent wizard resume

Process instances snapshot orchestrated work—wizard answers, step, status—and persist through storage so sessions survive restarts.

palm wizard start onboard
palm input Ada
palm instance list                    # note instance id

# Later, or in a new terminal:
palm process resume <instance_id>
palm input [e-mail]
# … continue through summary and commit

Shared StorageEngine across runtime lifetimes is required for cross-process resume (see DEVELOPMENT.md).

Durable filesystem storage (recommended for local dev and single-node deploys):

export PALM_STORAGE_BACKEND=filesystem
export PALM_DATA_DIR=./data   # optional; defaults to ./data

palm wizard start onboard
palm input Ada
# Restart the CLI — instances and definitions persist under ./data/
palm process resume <instance_id>

---

State snapshots (optional)

Palm can record point-in-time blackboard captures at selected job status transitions—useful for audit trails, debugging wizard flows, and future time-travel replay. Snapshots are stored on each ProcessInstance as a bounded ring buffer (state_snapshots[]). The feature is off by default.

Enable via environment:

export PALM_ENABLE_STATE_SNAPSHOT=true
export PALM_SNAPSHOT_ON_STATUS='["WAITING_FOR_INPUT","SUCCEEDED","FAILED"]'
export PALM_MAX_SNAPSHOTS_PER_INSTANCE=10

palm wizard start onboard
palm input Ada
palm instance snapshots <instance_id>   # inspect captured history

Enable in code:

from palm.app import PalmApp, PalmSettings

settings = PalmSettings(
    enable_state_snapshot=True,
    snapshot_on_status=["WAITING_FOR_INPUT", "SUCCEEDED"],
    max_snapshots_per_instance=5,
)
with PalmApp(settings) as app:
    app.create_runtime("embedded", autostart=True)
    job = app.submit_flow("onboard")
    snapshots = app.list_instance_snapshots(job.metadata["instance_id"])

Resume still uses the latest state_snapshot field (maintained by InstancePersistenceHook). Historical entries are for inspection—not replay yet. See ARCHITECTURE.md for middleware design and trade-offs.

---

Example flows

Definitions under examples/definitions/ auto-register at CLI startup.

Example	Command	Highlights
Onboarding	flow start onboard	Validation, summary + commit
Schema wizard	flow start schema-onboard	Flow + per-step schemas, scoped resume
Todo builder	flow start todo-builder	Collection step, dynamic lists, schemas
Parallel demo	flow start parallel-demo	Concurrent branches, merge, branch scopes
Data ingestion	flow start ingest-wizard	Resource action step, ETL companion flow
Approval	flow start approval	Multi-field validation, commit handler
Quick demo	flow start quick	Minimal wizard for resume experiments

palm process list
palm process submit data-ingestion
palm doctor    # shows flows with state schemas

Details: examples/README.md

---

CLI overview

Command	Description
palm / palm repl	Interactive REPL
palm doctor	Diagnostics: health, plugins, definitions, instances
palm version --full	Version, Python, registered patterns/providers/storages
palm process list | submit | resume	Definition catalog and lifecycle
palm instance list	Persisted instances
palm instance snapshots <id>	State snapshot history for an instance (when enabled)
palm flow start <flow>	Start any flow (wizard, parallel, …) — recommended
palm start <flow>	Shortcut for flow start
palm wizard start <flow>	Wizard-only shortcut (legacy alias)
palm input / palm back	Drive or rewind an active flow

Run palm --help for the full list.

---

Project structure

src/palm/
├── app/            # PalmApp orchestrator, settings, multi-runtime bootstrap
├── core/           # Pure engines (BT, orchestration, context, storage, …)
├── common/         # Shared coordination (plans, hooks, persistence, managers, StorageFactory)
├── instances/      # ProcessInstance + StateSnapshot models
├── definitions/    # FlowDefinition, ProcessDefinition
├── patterns/       # wizard, dag, etl (extensible)
├── providers/      # rest, graphql, postgres (extensible)
├── storages/       # memory, filesystem, postgres, mongodb (extensible)
└── runtimes/       # BaseRuntime, Embedded/Daemon/Server, CLI

examples/           # definitions/ + full_demo.py
SCOPE.md            # vision, scope, roadmap
ARCHITECTURE.md     # layers, middleware, BT model
archive/            # legacy + experimental (not imported)

---

Where Palm is headed

High-level direction (not all shipped yet). Full detail in SCOPE.md.

Theme	Direction
Runtimes	WebSocket surface, persistent plan registry, richer server auth
Middleware	Runtime-level auth/observability; optional BT guard nodes for step policy
Resources	Deeper ResourceEngine integration in patterns and commit handlers
Compute	KernelLeaf GPU nodes, resident kernels, dataset staging (Parquet → context → kernel → artifact)
Observability	Structured events, long-running job management

GPU batch prototypes live in archive/experimental/gpubatches/ as early R&D—not part of the supported API until promoted.

---
title: CPU vs GPU Execution Time
---
xychart
    title "CPU vs GPU Batch Processing Time"
    x-axis "Batch Size" ["32K", "65K", "131K", "262K"]
    y-axis "Time (seconds)" 0 --> 60
    line "CPU" [8.28, 14.37, 28.64, 57.14]
    line "GPU" [0.026, 0.051, 0.100, 0.200]

---

Architecture & contribution

Document	Contents
SCOPE.md	Vision, in/out of scope, roadmap, experimental areas
ARCHITECTURE.md	Layers, BT control flow, middleware model, engines
DEVELOPMENT.md	Setup, tests, adding patterns/backends
AGENTS.md	Rules for contributors and AI agents

just dev          # setup
just check        # lint + types + tests
just palm-doctor  # CLI health
just demo-full    # end-to-end script

---

Philosophy

🌴 Palm grows where the sun meets the sea.

Orchestration should balance structure with flexibility—automation with mindful human participation. Palm keeps the core small and truthful, puts people first in interactive flows, and grows capability through registries and nodes rather than monolithic middleware.

---

Migration

• 0.5.x → 0.6.0 — see MIGRATION-0.6.md for removed aliases (ExecutionBackend, EmbeddedMode, etc.)
• 0.3.x legacy — code under archive/ is reference-only; never import from archive/ in new work

---

License

MIT