A framework to build and run data pipelines and workflows.
Project description
NoETL
NoETL is an automation framework for orchestrating APIs, databases, and scripts using a declarative Playbook DSL.
Execution is standardized around an MCP-style tool model: consistent tool contracts, structured input/output, and a predictable lifecycle. From an MCP perspective, tools include API endpoints, database operations, and scripts/utilities NoETL orchestrates and optimizes them via playbooks.
With NoETL Gateway, playbooks can be deployed as a distributed backend: developers ship business logic as playbooks, and UIs/clients call stable endpoints without deploying dedicated microservices for each workflow.
Documentation
https://noetl.dev — user-facing site.
NoETL wiki — operator and
developer reference. Pages mirror the code tree under noetl/noetl.
Async batch acceptance and recovery references (on the wiki):
- Batch Events API —
POST /api/events/batchasync acceptance flow - Recovery: Auto-Resume — readiness-gated parent-execution restart at startup
- Command Reaper — runtime re-publish for orphaned / stranded commands
Distributed runtime components
Reference docs for the event-sourced, projection-backed runtime
(implements v2 distributed-runtime spec phases 0–2; spec lives in
noetl/docs at docs/features/noetl_distributed_runtime_spec.md):
- Event Store —
durable append-only event log (port + Postgres adapter),
EventRecordenvelope, optimistic concurrency viaexpected_version. - Projection Store — version-monotonic projection + snapshot store (port + Postgres adapter), query interface for replay state.
- Outbox — transactional
outbox publisher (
python -m noetl.outbox) that drainsnoetl.outboxto NATS with at-least-once retry/backoff. - Projector —
out-of-process projection worker (
python -m noetl.projector), durable NATS pull consumer, shard-stable, Prometheus metrics, replay-state folding shared with the in-process replay API.
EHDB Integration Contract
EHDB integration is disabled by default in this repository. The first NoETL-side surface is a feature-flagged contract only; it validates the execution-model boundary before any storage cutover exists.
Environment flags:
NOETL_EHDB_ENABLED=trueturns on contract validation.NOETL_EHDB_MODE=control_plane|local_referenceselects either control-plane embedding or local reference readiness mode.NOETL_EHDB_CLIENT_ROLE=gateway|api|server|worker|playbook|systemdeclares the caller role.NOETL_EHDB_CAPABILITIES=control_planeis the only capability accepted for gateway/API/server control-plane embedding. Worker, playbook, and system local-reference configs default to explicit data-plane capabilities unless narrowed with a comma-separated list.NOETL_EHDB_LOCAL_REFERENCE_LOG=/path/to/ehdb.jsonlprovides the explicit local event-log path for the reference mode.NOETL_EHDB_HELPER_BIN=/path/to/helperis required only when a worker/playbook asks NoETL to build a local-reference helper invocation plan.
Gateway/API/server roles are accepted only in explicit control_plane
mode with the control_plane capability. They are still rejected for
local_reference and any data-plane capability. Gateway remains the
gatekeeper; workers remain atomic compute; playbooks remain ephemeral
blueprints; shared cache remains a state vehicle; the event log remains
the source of truth. This contract does not connect to EHDB, replace
PostgreSQL/NATS/object stores, add a gateway route, or start a
persistent per-tenant process.
noetl.core.ehdb_control_plane.ehdb_control_plane_from_env builds the
planning-only descriptor for gateway/API/server control-plane embedding.
Disabled configuration returns None; explicit control_plane
configuration returns a ControlPlaneEhdbEmbedding carrying the caller
role, the control_plane capability, and exportable runtime
environment. The descriptor does not create an adapter, open logs,
connect to EHDB, or perform storage operations.
noetl.core.ehdb_surface.ehdb_surface_from_env is the common
side-effect-free selector. It returns None when EHDB is disabled,
returns the control-plane descriptor for gateway/API/server
control_plane configs, and returns the local-reference adapter for
worker/playbook/system data-plane configs. The selected surface exposes
role, mode, capabilities, and runtime env without opening logs,
executing helpers, or importing EHDB.
noetl.core.ehdb_adapter.ehdb_adapter_from_env builds the disabled-by-
default adapter descriptor behind that contract. Disabled configuration
returns None; gateway/API/server control_plane configuration also
returns None because it has no data-plane helper. Worker/playbook
local_reference configuration returns a LocalReferenceEhdbAdapter
carrying the explicit event-log path, data-plane capability set, and
exportable runtime environment for future EHDB helper calls. The adapter
does not open logs, connect to EHDB, or perform storage operations.
noetl.core.ehdb_adapter.ehdb_helper_invocation_from_env builds the
next planning surface for those helper calls. Disabled configuration
returns None; enabled worker/playbook configuration requires an
explicit helper executable and returns deterministic argv plus EHDB
runtime env that can be merged into a subprocess environment. The
invocation plan is immutable and side-effect-free; it does not execute a
subprocess, import EHDB, open logs, connect to storage, or add
gateway/server data paths.
noetl.core.ehdb_adapter.ehdb_local_reference_summary_invocation_from_env
builds the first concrete helper command:
ehdb-local-reference summary --log <path>. Worker/playbook contexts
may pass that invocation to
noetl.core.ehdb_adapter.execute_ehdb_helper_json, which runs the
helper without a shell, captures stdout/stderr, enforces a timeout, and
decodes a JSON object. Disabled configuration returns None, and
gateway/API/server local-reference roles remain rejected by the
contract. This runner is for bounded local diagnostics and integration
tests; it does not import Rust EHDB, open storage from the gateway,
replace platform dependencies, or create persistent per-tenant
processes.
Helper discovery prefers explicit NOETL_EHDB_HELPER_BIN, then
ehdb-local-reference on PATH, then image/runtime paths
/usr/local/bin/ehdb-local-reference and
/opt/noetl/bin/ehdb-local-reference, then the ai-meta sibling EHDB
development build outputs under ../ehdb/target/{release,debug}.
scripts/smoke_ehdb_local_reference_summary.py exercises this path by
running summary --log <path> and validating the returned JSON summary
shape.
Repository model (ai-meta driven)
NoETL development is now coordinated through the ai-meta repository:
- Orchestration/meta: https://github.com/noetl/ai-meta
- Server: https://github.com/noetl/server
- Worker: https://github.com/noetl/worker
- Gateway: https://github.com/noetl/gateway
- CLI: https://github.com/noetl/cli
- Tools: https://github.com/noetl/tools
- End-to-end fixtures: https://github.com/noetl/e2e
- Ops/automation: https://github.com/noetl/ops
- Docs: https://github.com/noetl/docs
ai-meta tracks all component repositories as Git submodules and is the primary place to
coordinate cross-repo changes, pointer bumps, and release choreography.
End-to-end integration playbooks, fixture payloads, local credential templates, notebooks,
and Gateway UI test fixtures live in noetl/e2e.
Distribution channels
- PyPI
noetl(Python): https://pypi.org/project/noetl/
- Rust components
- CLI repo: https://github.com/noetl/cli
- Gateway repo: https://github.com/noetl/gateway
- Server repo: https://github.com/noetl/server
- Worker repo: https://github.com/noetl/worker
- Tools repo: https://github.com/noetl/tools
- APT (Debian/Ubuntu)
- Homebrew
License
Dual License — see LICENSE for details.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file noetl-4.20.0.tar.gz.
File metadata
- Download URL: noetl-4.20.0.tar.gz
- Upload date:
- Size: 887.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.9.30 {"installer":{"name":"uv","version":"0.9.30","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
5b013a128119395efd27934256933f6544674a5bdf12194f4f9477d2c52a6f19
|
|
| MD5 |
2c0d9f150c140a8dc8fdbd0a70f34d9d
|
|
| BLAKE2b-256 |
4642f4490fc0907569d5fb46be90e80ac3ded1181e6d50dd7fea87b9dfde9380
|
File details
Details for the file noetl-4.20.0-py3-none-any.whl.
File metadata
- Download URL: noetl-4.20.0-py3-none-any.whl
- Upload date:
- Size: 989.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.9.30 {"installer":{"name":"uv","version":"0.9.30","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
830e664b7f2db3858dd375c3e4972cb23cc6cc3f039fc9fcfa768a65ee8e4e9a
|
|
| MD5 |
072958b5a334e2477b39c26cba18c25e
|
|
| BLAKE2b-256 |
7d551b1ce16122b6088171b2f413d9714c7e6ea89ca4993b16cd3e204ec94537
|
