Ergon Framework — Python SDK
The official Python implementation of the Ergon Framework—a transaction-first, observability-native execution engine for building high-throughput automation pipelines.
Installation
pip install ergon-framework-python
Connector integrations are optional extras, so a core installation does not pull in client libraries that your application does not use:
pip install 'ergon-framework-python[rabbitmq]'
pip install 'ergon-framework-python[sqs,postgres]'
pip install 'ergon-framework-python[outlook]'
pip install 'ergon-framework-python[all]'
Available extras are rabbitmq, sqs, postgres, pipefy, excel,
ergon-platform, nylas, and outlook.
Outlook / Microsoft Graph
pip install 'ergon-framework-python[outlook]'
The Outlook connector talks to Microsoft Graph with Entra application
credentials. Consuming messages and attachments needs Mail.Read; marking,
categorizing, moving, and deleting need Mail.ReadWrite; send, reply, and
forward need Mail.Send. Graph $filter and $search cannot be combined.
from ergon.connector.outlook import (
AsyncOutlookGraphConnector,
OutlookAckActionConfig,
OutlookConsumerConfig,
OutlookGraphClient,
OutlookMessageFilter,
)
client = OutlookGraphClient(
tenant_id="...",
client_id="...",
client_secret="...",
user_email="mailbox@example.com",
)
consumer = OutlookConsumerConfig(
folder_id="Inbox",
filter=OutlookMessageFilter(unread_only=True, has_attachments=True),
download_attachments=True,
ack_config=OutlookAckActionConfig(mark_as_read=True),
)
Runnable example: examples/outlook. Module reference: Outlook connector README.
Local development
Install in editable mode:
pip install -e /path/to/ergon-framework/sdks/python
📖 Full Getting Started Guide — Complete setup, project configuration, and task registration
Quick Start
1. Define a Consumer Task
Create a task by inheriting from ConsumerTask and implementing process_transaction:
from typing import Any
from ergon.task.mixins import ConsumerTask
from ergon.connector import Transaction
class OrderProcessor(ConsumerTask):
"""Consumes order transactions and processes them."""
def process_transaction(self, transaction: Transaction) -> Any:
# Pure business logic—no I/O, no retries, no protocol details
order = transaction.payload
total = order["quantity"] * order["unit_price"]
return {"order_id": order["id"], "total": total, "status": "processed"}
def handle_process_success(self, transaction: Transaction, result: Any):
# Called after successful processing
self.output_connector.dispatch_transactions([
Transaction(id=f"out-{transaction.id}", payload=result, metadata={})
])
def handle_process_exception(self, transaction: Transaction, exc: Exception):
# Called when processing fails (after all retries exhausted)
self.dlq_connector.dispatch_transactions([transaction])
2. Create a Connector
Connectors bridge external systems to the transaction interface:
from typing import List
from ergon.connector import Connector, Transaction
import uuid
class RabbitMQConnector(Connector):
def __init__(self, queue_name: str, connection_url: str):
self.queue_name = queue_name
# Initialize your RabbitMQ client here
self.client = create_rabbitmq_client(connection_url)
def fetch_transactions(self, batch_size: int, **kwargs) -> List[Transaction]:
messages = self.client.consume(self.queue_name, max_messages=batch_size)
return [
Transaction(
id=str(uuid.uuid4()),
payload=msg.body,
metadata={"delivery_tag": msg.delivery_tag, "routing_key": msg.routing_key}
)
for msg in messages
]
def dispatch_transactions(self, transactions: List[Transaction], **kwargs):
for tx in transactions:
self.client.publish(self.queue_name, tx.payload)
3. Configure and Run
Use TaskConfig to wire everything together:
from ergon.task import TaskConfig, runner, policies
from ergon.connector import ConnectorConfig
# Configure the consumer policy
consumer_policy = policies.ConsumerPolicy()
consumer_policy.name = "consumer"
consumer_policy.loop.concurrency.value = 5 # Process 5 transactions in parallel
consumer_policy.loop.batch.size = 10 # Fetch 10 at a time
consumer_policy.loop.streaming = True # Keep polling for new messages
consumer_policy.process.retry.max_attempts = 3 # Retry failed processing 3 times
consumer_policy.process.retry.backoff = 1.0 # Start with 1s backoff
consumer_policy.process.retry.backoff_multiplier = 2.0 # Double each retry
# Create the task configuration
config = TaskConfig(
name="order-processor",
task=OrderProcessor,
max_workers=1,
connectors={
"input": ConnectorConfig(
connector=RabbitMQConnector,
kwargs={"queue_name": "orders", "connection_url": "amqp://localhost"}
),
"output": ConnectorConfig(
connector=RabbitMQConnector,
kwargs={"queue_name": "processed-orders", "connection_url": "amqp://localhost"}
),
},
policies=[consumer_policy],
)
# Run the task
if __name__ == "__main__":
runner.run(config)
The framework handles the loop, retry logic, telemetry, and graceful shutdown automatically.
The Transaction Object
In the Python SDK, transactions are implemented as immutable Pydantic models:
from ergon.connector import Transaction
# Transactions are created by Connectors when fetching data
tx = Transaction(
id="tx-001", # Unique identifier for tracing
payload={"order_id": 123, "amount": 99.99}, # The actual data
metadata={"source": "rabbitmq", "queue": "orders"} # Contextual info
)
Transactions are frozen (immutable) once created via Pydantic's frozen=True configuration.
Task Classes
The Python SDK provides ready-to-use task classes:
from ergon.task.mixins import (
ConsumerTask, # Sync consumer
ProducerTask, # Sync producer
HybridTask, # Sync consumer + producer
AsyncConsumerTask, # Async consumer
AsyncProducerTask, # Async producer
AsyncHybridTask, # Async consumer + producer
)
Synchronous Tasks
Best for CPU-bound processing or legacy libraries:
from ergon.task.mixins import ConsumerTask
class SyncProcessor(ConsumerTask):
def process_transaction(self, transaction):
# Runs in ThreadPoolExecutor
return heavy_computation(transaction.payload)
- Concurrency controlled by
policy.loop.concurrency.value - Each transaction processed in its own thread
- Multi-process scaling via
max_workers > 1
Asynchronous Tasks
Best for I/O-bound workloads with high concurrency:
from ergon.task.mixins import AsyncConsumerTask
class AsyncProcessor(AsyncConsumerTask):
async def process_transaction(self, transaction):
# Runs in asyncio event loop
result = await self.http_client.post(transaction.payload)
return result
- Concurrency controlled by
asyncio.Semaphore - Thousands of concurrent operations with minimal overhead
- Perfect for API calls, database queries, network I/O
Dependency Injection
Tasks receive all dependencies automatically via the TaskMeta metaclass:
class EnrichmentTask(ConsumerTask):
def process_transaction(self, transaction: Transaction) -> Any:
# Access connectors as self.{name}_connector
count = self.input_connector.get_transactions_count()
# Access services as self.{name}_service
enriched = self.openai_service.complete(transaction.payload["text"])
# Access policies as self.{name}_policy
timeout = self.consumer_policy.loop.timeout
return {"enriched": enriched, "pending": count}
Configure services in TaskConfig:
from ergon.connector import ServiceConfig
config = TaskConfig(
name="enrichment-task",
task=EnrichmentTask,
connectors={"input": input_connector_config},
services={
"openai": ServiceConfig(
service=OpenAIService,
kwargs={"api_key": "sk-...", "model": "gpt-4"}
)
},
policies=[consumer_policy],
)
Policy Configuration
Policies are created as instances and configured by setting properties directly:
from ergon.task import policies
# Consumer policy with full configuration
policy = policies.ConsumerPolicy()
policy.name = "my-consumer"
# Loop behavior
policy.loop.concurrency.value = 10 # Parallel transaction processing
policy.loop.batch.size = 50 # Transactions per fetch
policy.loop.streaming = True # Continuous polling mode
policy.loop.timeout = 3600 # Max loop duration (seconds)
policy.loop.limit = 1000 # Max transactions to process
# Empty queue behavior (for streaming mode)
policy.loop.empty_queue.backoff = 1.0
policy.loop.empty_queue.backoff_multiplier = 2.0
policy.loop.empty_queue.backoff_cap = 30.0
# Per-step retry configuration
policy.fetch.retry.max_attempts = 5
policy.fetch.retry.timeout = 30
policy.process.retry.max_attempts = 3
policy.process.retry.backoff = 1.0
policy.process.retry.backoff_multiplier = 2.0
policy.process.retry.backoff_cap = 60.0
policy.success.retry.max_attempts = 2
policy.exception.retry.max_attempts = 2
Observability
The Python SDK integrates OpenTelemetry natively:
from ergon.telemetry import logging, tracing, metrics
config = TaskConfig(
name="observed-task",
task=MyTask,
connectors={...},
logging=logging.LoggingConfig(
level="INFO",
handlers=[
logging.ConsoleLogHandler(),
logging.OTLPLogHandler(endpoint="http://collector:4317"),
]
),
tracing=tracing.TracingConfig(
processors=[
tracing.SpanProcessor(
processor=tracing.BatchSpanProcessor,
exporters=[tracing.OTLPSpanExporter(endpoint="http://collector:4317")]
)
]
),
metrics=metrics.MetricsConfig(
readers=[
metrics.MetricReader(
reader=metrics.PeriodicExportingMetricReader,
exporters=[metrics.OTLPMetricExporter(endpoint="http://collector:4317")]
)
]
),
)
Logging
- Structured JSON logging via
python-json-logger - Automatic
trace_idandspan_idinjection - Multiple handlers: Console, File, RotatingFile, OTLP
Tracing
- Hierarchical spans: Task → Batch → Transaction → Attempt
- Full context propagation across async boundaries
- Export to Jaeger, Tempo, or any OTLP-compatible backend
Metrics
- Push-based via
PeriodicExportingMetricReader - Automatic resource attributes (task name, host, PID, execution ID)
- Export to Prometheus, OTLP, or console
Scaling
| Axis | Mechanism | Configuration |
|---|---|---|
| Process | ProcessPoolExecutor |
TaskConfig.max_workers |
| Thread | ThreadPoolExecutor |
policy.loop.concurrency.value |
| Async | asyncio.Semaphore |
policy.loop.concurrency.value |
| Connector | External system partitioning | Service-specific (consumer groups, shards) |
# Multi-process scaling (sync tasks only)
config = TaskConfig(
name="scaled-task",
task=MyTask,
max_workers=4, # Spawn 4 isolated worker processes
connectors={...},
)
Project Structure
After scaffolding with ergon init:
my-project/
├── main.py # Application entry point
├── _observability/ # Telemetry infrastructure configs
├── connectors/ # Custom connectors and services
│ └── {connector_name}/ # One submodule per connector
│ ├── connector.py # Transaction interface
│ └── service.py # Protocol mechanics
└── tasks/ # Task definitions and shared modules
├── settings.py # Global configs (connectors, services, telemetry)
├── constants.py # Global constants and enums
├── schemas.py # Shared Pydantic models
├── exceptions.py # Shared exception classes
├── helpers.py # Shared utility functions
└── {task_name}/ # Per-task submodule
├── task.py # Task implementation
├── config.py # TaskConfig definition
├── schemas.py # Task-specific models
├── exceptions.py # Task-specific exceptions
└── helpers.py # Task-specific utilities
📖 Full Project Structure Guide — Detailed documentation on organizing connectors, tasks, and shared modules.
Documentation
Python SDK Guides
- Getting Started — Installation, project setup, and running your first task
- CLI Reference — Commands, options, and exit codes
- Project Structure Guide — How to organize connectors, tasks, and shared modules
Framework Concepts
- Framework Architecture — Full system specification and design philosophy
- Transaction Abstraction — Understanding atomicity rules
- Task Module — Mixins, lifecycles, and execution modes
- Connector Module — Building integration boundaries
- Service Module — Protocol engineering and reliability
- Telemetry Module — Configuring OTel logs, metrics, and traces
License
This project is licensed under the MIT License.
Metadata
Release files for ergon-framework-python 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ergon_framework_python-0.3.0.tar.gz | 135.5 kB | Without provenance | Details for ergon_framework_python-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ergon_framework_python-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for ergon_framework_python-0.3.0-py3-none-any.whl |
Total release size: 314.4 kB
Release files / ergon_framework_python-0.3.0.tar.gz
| Download URL | ergon_framework_python-0.3.0.tar.gz |
|---|---|
| Size | 135.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2361607fe92863efe3badea2f063cf493d886191700016fcf53f6ba231d52022
|
|
BLAKE2b-256 checksum How to use checksums |
6318a30426765a7e2c319f05a40e138d481b6ce555729f7b1bfe4067f4bcec2e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / ergon_framework_python-0.3.0-py3-none-any.whl
| Download URL | ergon_framework_python-0.3.0-py3-none-any.whl |
|---|---|
| Size | 178.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
41d29df1ae15404a4047eb0bf1d97353286f0b5a077f902afd88a10c09154bd7
|
|
BLAKE2b-256 checksum How to use checksums |
56f660ac44c56064a98e6c36a78bcdb6165e51d6ac14f9f8611a2dcdd0627cd4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for ergon-framework-python 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.