A framework to build and run data pipelines and workflows.
Project description
NoETL
NoETL is an automation framework for orchestrating APIs, databases, and scripts using a declarative Playbook DSL.
Execution is standardized around an MCP-style tool model: consistent tool contracts, structured input/output, and a predictable lifecycle. From an MCP perspective, tools include API endpoints, database operations, and scripts/utilities NoETL orchestrates and optimizes them via playbooks.
With NoETL Gateway, playbooks can be deployed as a distributed backend: developers ship business logic as playbooks, and UIs/clients call stable endpoints without deploying dedicated microservices for each workflow.
Documentation
https://noetl.dev — user-facing site.
NoETL wiki — operator and
developer reference. Pages mirror the code tree under noetl/noetl.
Async batch acceptance and recovery references (on the wiki):
- Batch Events API —
POST /api/events/batchasync acceptance flow - Recovery: Auto-Resume — readiness-gated parent-execution restart at startup
- Command Reaper — runtime re-publish for orphaned / stranded commands
Distributed runtime components
Reference docs for the event-sourced, projection-backed runtime
(implements v2 distributed-runtime spec phases 0–2; spec lives in
noetl/docs at docs/features/noetl_distributed_runtime_spec.md):
- Event Store —
durable append-only event log (port + Postgres adapter),
EventRecordenvelope, optimistic concurrency viaexpected_version. - Projection Store — version-monotonic projection + snapshot store (port + Postgres adapter), query interface for replay state.
- Outbox — transactional
outbox publisher (
python -m noetl.outbox) that drainsnoetl.outboxto NATS with at-least-once retry/backoff. - Projector —
out-of-process projection worker (
python -m noetl.projector), durable NATS pull consumer, shard-stable, Prometheus metrics, replay-state folding shared with the in-process replay API.
EHDB Integration Contract
EHDB integration is disabled by default in this repository. The first NoETL-side surface is a feature-flagged contract only; it validates the execution-model boundary before any storage cutover exists.
Environment flags:
NOETL_EHDB_ENABLED=trueturns on contract validation.NOETL_EHDB_MODE=local_referenceselects the local reference readiness mode.NOETL_EHDB_CLIENT_ROLE=worker|playbookdeclares the caller role.NOETL_EHDB_LOCAL_REFERENCE_LOG=/path/to/ehdb.jsonlprovides the explicit local event-log path for the reference mode.
Gateway/server roles are rejected when EHDB is enabled. Gateway remains the gatekeeper; workers remain atomic compute; playbooks remain ephemeral blueprints; shared cache remains a state vehicle; the event log remains the source of truth. This contract does not connect to EHDB, replace PostgreSQL/NATS/object stores, add a gateway route, or start a persistent per-tenant process.
noetl.core.ehdb_adapter.ehdb_adapter_from_env builds the disabled-by-
default adapter descriptor behind that contract. Disabled configuration
returns None; worker/playbook local_reference configuration returns
a LocalReferenceEhdbAdapter carrying the explicit event-log path and
exportable runtime environment for future EHDB helper calls. The adapter
does not open logs, connect to EHDB, or perform storage operations.
Repository model (ai-meta driven)
NoETL development is now coordinated through the ai-meta repository:
- Orchestration/meta: https://github.com/noetl/ai-meta
- Server: https://github.com/noetl/server
- Worker: https://github.com/noetl/worker
- Gateway: https://github.com/noetl/gateway
- CLI: https://github.com/noetl/cli
- Tools: https://github.com/noetl/tools
- End-to-end fixtures: https://github.com/noetl/e2e
- Ops/automation: https://github.com/noetl/ops
- Docs: https://github.com/noetl/docs
ai-meta tracks all component repositories as Git submodules and is the primary place to
coordinate cross-repo changes, pointer bumps, and release choreography.
End-to-end integration playbooks, fixture payloads, local credential templates, notebooks,
and Gateway UI test fixtures live in noetl/e2e.
Distribution channels
- PyPI
noetl(Python): https://pypi.org/project/noetl/
- Rust components
- CLI repo: https://github.com/noetl/cli
- Gateway repo: https://github.com/noetl/gateway
- Server repo: https://github.com/noetl/server
- Worker repo: https://github.com/noetl/worker
- Tools repo: https://github.com/noetl/tools
- APT (Debian/Ubuntu)
- Homebrew
License
Dual License — see LICENSE for details.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file noetl-4.14.0.tar.gz.
File metadata
- Download URL: noetl-4.14.0.tar.gz
- Upload date:
- Size: 882.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.9.30 {"installer":{"name":"uv","version":"0.9.30","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
bbbf89038d66621266838a0d6a5353d34fc58a4298b9354af5bee1682ba6d8c7
|
|
| MD5 |
0698a91a8f2adccdaef2e42352417efb
|
|
| BLAKE2b-256 |
d3aa6e3601a3a0498639ee29d58b60c31d183ce7abed8b7a5903a027426aa130
|
File details
Details for the file noetl-4.14.0-py3-none-any.whl.
File metadata
- Download URL: noetl-4.14.0-py3-none-any.whl
- Upload date:
- Size: 983.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.9.30 {"installer":{"name":"uv","version":"0.9.30","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
463a349f69939800f48b5d08bb98c500a65324bd66c84eba1a2a612b89ca7e14
|
|
| MD5 |
7078eed231f578b1f8f2957a0a3cdf66
|
|
| BLAKE2b-256 |
dbe47bab51aedef3c564dee4807df089ad33d04697fbebe43ab36f1a48ec7cc2
|
