Skip to main content

Ergon Framework — Python SDK

Python 3.10+ License: MIT Code Style: Ruff

The official Python implementation of the Ergon Framework—a transaction-first, observability-native execution engine for building high-throughput automation pipelines.


Installation

pip install ergon-framework-python

Connector integrations are optional extras, so a core installation does not pull in client libraries that your application does not use:

pip install 'ergon-framework-python[rabbitmq]'
pip install 'ergon-framework-python[sqs,postgres]'
pip install 'ergon-framework-python[outlook]'
pip install 'ergon-framework-python[all]'

Available extras are rabbitmq, sqs, postgres, pipefy, excel, ergon-platform, nylas, and outlook.

Outlook / Microsoft Graph

pip install 'ergon-framework-python[outlook]'

The Outlook connector talks to Microsoft Graph with Entra application credentials. Consuming messages and attachments needs Mail.Read; marking, categorizing, moving, and deleting need Mail.ReadWrite; send, reply, and forward need Mail.Send. Graph $filter and $search cannot be combined.

from ergon.connector.outlook import (
    AsyncOutlookGraphConnector,
    OutlookAckActionConfig,
    OutlookConsumerConfig,
    OutlookGraphClient,
    OutlookMessageFilter,
)

client = OutlookGraphClient(
    tenant_id="...",
    client_id="...",
    client_secret="...",
    user_email="mailbox@example.com",
)
consumer = OutlookConsumerConfig(
    folder_id="Inbox",
    filter=OutlookMessageFilter(unread_only=True, has_attachments=True),
    download_attachments=True,
    ack_config=OutlookAckActionConfig(mark_as_read=True),
)

Runnable example: examples/outlook. Module reference: Outlook connector README.

Local development

Install in editable mode:

pip install -e /path/to/ergon-framework/sdks/python

📖 Full Getting Started Guide — Complete setup, project configuration, and task registration


Quick Start

1. Define a Consumer Task

Create a task by inheriting from ConsumerTask and implementing process_transaction:

from typing import Any
from ergon.task.mixins import ConsumerTask
from ergon.connector import Transaction

class OrderProcessor(ConsumerTask):
    """Consumes order transactions and processes them."""

    def process_transaction(self, transaction: Transaction) -> Any:
        # Pure business logic—no I/O, no retries, no protocol details
        order = transaction.payload
        total = order["quantity"] * order["unit_price"]
        return {"order_id": order["id"], "total": total, "status": "processed"}

    def handle_process_success(self, transaction: Transaction, result: Any):
        # Called after successful processing
        self.output_connector.dispatch_transactions([
            Transaction(id=f"out-{transaction.id}", payload=result, metadata={})
        ])

    def handle_process_exception(self, transaction: Transaction, exc: Exception):
        # Called when processing fails (after all retries exhausted)
        self.dlq_connector.dispatch_transactions([transaction])

2. Create a Connector

Connectors bridge external systems to the transaction interface:

from typing import List
from ergon.connector import Connector, Transaction
import uuid

class RabbitMQConnector(Connector):
    def __init__(self, queue_name: str, connection_url: str):
        self.queue_name = queue_name
        # Initialize your RabbitMQ client here
        self.client = create_rabbitmq_client(connection_url)

    def fetch_transactions(self, batch_size: int, **kwargs) -> List[Transaction]:
        messages = self.client.consume(self.queue_name, max_messages=batch_size)
        return [
            Transaction(
                id=str(uuid.uuid4()),
                payload=msg.body,
                metadata={"delivery_tag": msg.delivery_tag, "routing_key": msg.routing_key}
            )
            for msg in messages
        ]

    def dispatch_transactions(self, transactions: List[Transaction], **kwargs):
        for tx in transactions:
            self.client.publish(self.queue_name, tx.payload)

3. Configure and Run

Use TaskConfig to wire everything together:

from ergon.task import TaskConfig, runner, policies
from ergon.connector import ConnectorConfig

# Configure the consumer policy
consumer_policy = policies.ConsumerPolicy()
consumer_policy.name = "consumer"
consumer_policy.loop.concurrency.value = 5      # Process 5 transactions in parallel
consumer_policy.loop.batch.size = 10            # Fetch 10 at a time
consumer_policy.loop.streaming = True           # Keep polling for new messages
consumer_policy.process.retry.max_attempts = 3  # Retry failed processing 3 times
consumer_policy.process.retry.backoff = 1.0     # Start with 1s backoff
consumer_policy.process.retry.backoff_multiplier = 2.0  # Double each retry

# Create the task configuration
config = TaskConfig(
    name="order-processor",
    task=OrderProcessor,
    max_workers=1,
    connectors={
        "input": ConnectorConfig(
            connector=RabbitMQConnector,
            kwargs={"queue_name": "orders", "connection_url": "amqp://localhost"}
        ),
        "output": ConnectorConfig(
            connector=RabbitMQConnector,
            kwargs={"queue_name": "processed-orders", "connection_url": "amqp://localhost"}
        ),
    },
    policies=[consumer_policy],
)

# Run the task
if __name__ == "__main__":
    runner.run(config)

The framework handles the loop, retry logic, telemetry, and graceful shutdown automatically.


The Transaction Object

In the Python SDK, transactions are implemented as immutable Pydantic models:

from ergon.connector import Transaction

# Transactions are created by Connectors when fetching data
tx = Transaction(
    id="tx-001",                           # Unique identifier for tracing
    payload={"order_id": 123, "amount": 99.99},  # The actual data
    metadata={"source": "rabbitmq", "queue": "orders"}  # Contextual info
)

Transactions are frozen (immutable) once created via Pydantic's frozen=True configuration.


Task Classes

The Python SDK provides ready-to-use task classes:

from ergon.task.mixins import (
    ConsumerTask,      # Sync consumer
    ProducerTask,      # Sync producer
    HybridTask,        # Sync consumer + producer
    AsyncConsumerTask, # Async consumer
    AsyncProducerTask, # Async producer
    AsyncHybridTask,   # Async consumer + producer
)

Synchronous Tasks

Best for CPU-bound processing or legacy libraries:

from ergon.task.mixins import ConsumerTask

class SyncProcessor(ConsumerTask):
    def process_transaction(self, transaction):
        # Runs in ThreadPoolExecutor
        return heavy_computation(transaction.payload)
  • Concurrency controlled by policy.loop.concurrency.value
  • Each transaction processed in its own thread
  • Multi-process scaling via max_workers > 1

Asynchronous Tasks

Best for I/O-bound workloads with high concurrency:

from ergon.task.mixins import AsyncConsumerTask

class AsyncProcessor(AsyncConsumerTask):
    async def process_transaction(self, transaction):
        # Runs in asyncio event loop
        result = await self.http_client.post(transaction.payload)
        return result
  • Concurrency controlled by asyncio.Semaphore
  • Thousands of concurrent operations with minimal overhead
  • Perfect for API calls, database queries, network I/O

Dependency Injection

Tasks receive all dependencies automatically via the TaskMeta metaclass:

class EnrichmentTask(ConsumerTask):
    def process_transaction(self, transaction: Transaction) -> Any:
        # Access connectors as self.{name}_connector
        count = self.input_connector.get_transactions_count()
      
        # Access services as self.{name}_service
        enriched = self.openai_service.complete(transaction.payload["text"])
      
        # Access policies as self.{name}_policy
        timeout = self.consumer_policy.loop.timeout
      
        return {"enriched": enriched, "pending": count}

Configure services in TaskConfig:

from ergon.connector import ServiceConfig

config = TaskConfig(
    name="enrichment-task",
    task=EnrichmentTask,
    connectors={"input": input_connector_config},
    services={
        "openai": ServiceConfig(
            service=OpenAIService,
            kwargs={"api_key": "sk-...", "model": "gpt-4"}
        )
    },
    policies=[consumer_policy],
)

Policy Configuration

Policies are created as instances and configured by setting properties directly:

from ergon.task import policies

# Consumer policy with full configuration
policy = policies.ConsumerPolicy()
policy.name = "my-consumer"

# Loop behavior
policy.loop.concurrency.value = 10      # Parallel transaction processing
policy.loop.batch.size = 50             # Transactions per fetch
policy.loop.streaming = True            # Continuous polling mode
policy.loop.timeout = 3600              # Max loop duration (seconds)
policy.loop.limit = 1000                # Max transactions to process

# Empty queue behavior (for streaming mode)
policy.loop.empty_queue.backoff = 1.0
policy.loop.empty_queue.backoff_multiplier = 2.0
policy.loop.empty_queue.backoff_cap = 30.0

# Per-step retry configuration
policy.fetch.retry.max_attempts = 5
policy.fetch.retry.timeout = 30

policy.process.retry.max_attempts = 3
policy.process.retry.backoff = 1.0
policy.process.retry.backoff_multiplier = 2.0
policy.process.retry.backoff_cap = 60.0

policy.success.retry.max_attempts = 2
policy.exception.retry.max_attempts = 2

Observability

The Python SDK integrates OpenTelemetry natively:

from ergon.telemetry import logging, tracing, metrics

config = TaskConfig(
    name="observed-task",
    task=MyTask,
    connectors={...},
    logging=logging.LoggingConfig(
        level="INFO",
        handlers=[
            logging.ConsoleLogHandler(),
            logging.OTLPLogHandler(endpoint="http://collector:4317"),
        ]
    ),
    tracing=tracing.TracingConfig(
        processors=[
            tracing.SpanProcessor(
                processor=tracing.BatchSpanProcessor,
                exporters=[tracing.OTLPSpanExporter(endpoint="http://collector:4317")]
            )
        ]
    ),
    metrics=metrics.MetricsConfig(
        readers=[
            metrics.MetricReader(
                reader=metrics.PeriodicExportingMetricReader,
                exporters=[metrics.OTLPMetricExporter(endpoint="http://collector:4317")]
            )
        ]
    ),
)

Logging

  • Structured JSON logging via python-json-logger
  • Automatic trace_id and span_id injection
  • Multiple handlers: Console, File, RotatingFile, OTLP

Tracing

  • Hierarchical spans: Task → Batch → Transaction → Attempt
  • Full context propagation across async boundaries
  • Export to Jaeger, Tempo, or any OTLP-compatible backend

Metrics

  • Push-based via PeriodicExportingMetricReader
  • Automatic resource attributes (task name, host, PID, execution ID)
  • Export to Prometheus, OTLP, or console

Scaling

Axis Mechanism Configuration
Process ProcessPoolExecutor TaskConfig.max_workers
Thread ThreadPoolExecutor policy.loop.concurrency.value
Async asyncio.Semaphore policy.loop.concurrency.value
Connector External system partitioning Service-specific (consumer groups, shards)
# Multi-process scaling (sync tasks only)
config = TaskConfig(
    name="scaled-task",
    task=MyTask,
    max_workers=4,  # Spawn 4 isolated worker processes
    connectors={...},
)

Project Structure

After scaffolding with ergon init:

my-project/
├── main.py                  # Application entry point
├── _observability/          # Telemetry infrastructure configs
├── connectors/              # Custom connectors and services
│   └── {connector_name}/    # One submodule per connector
│       ├── connector.py     # Transaction interface
│       └── service.py       # Protocol mechanics
└── tasks/                   # Task definitions and shared modules
    ├── settings.py          # Global configs (connectors, services, telemetry)
    ├── constants.py         # Global constants and enums
    ├── schemas.py           # Shared Pydantic models
    ├── exceptions.py        # Shared exception classes
    ├── helpers.py           # Shared utility functions
    └── {task_name}/         # Per-task submodule
        ├── task.py          # Task implementation
        ├── config.py        # TaskConfig definition
        ├── schemas.py       # Task-specific models
        ├── exceptions.py    # Task-specific exceptions
        └── helpers.py       # Task-specific utilities

📖 Full Project Structure Guide — Detailed documentation on organizing connectors, tasks, and shared modules.


Documentation

Python SDK Guides

Framework Concepts


License

This project is licensed under the MIT License.


Metadata

Release files for ergon-framework-python 0.2.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ergon-framework-python 0.2.5
File Size Uploaded Provenance Actions
ergon_framework_python-0.2.5.tar.gz 131.7 kB Without provenance Details for ergon_framework_python-0.2.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ergon-framework-python 0.2.5
File Interpreter ABI Platform Actions
ergon_framework_python-0.2.5-py3-none-any.whl Python 3 none any Details for ergon_framework_python-0.2.5-py3-none-any.whl

Total release size: 305.7 kB

Release files / ergon_framework_python-0.2.5.tar.gz

Download URL ergon_framework_python-0.2.5.tar.gz
Size 131.7 kB
Tags Source
SHA-256 checksum
How to use checksums
9bac76665e6563c9be33c709300f2dd7d774f18944fd311728fc870f9460d694
BLAKE2b-256 checksum
How to use checksums
6ebca8d60897a208cb1b5b6840eece46f170c4a18af22fdf8765a1250fd6ebe4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / ergon_framework_python-0.2.5-py3-none-any.whl

Download URL ergon_framework_python-0.2.5-py3-none-any.whl
Size 174.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
dfe5afbc84ed833b94a7acef2424871144de1b9a51e68b5a95664504061663a3
BLAKE2b-256 checksum
How to use checksums
1ce177a0cd08c7001b0f1ec832d819788aeedc06f72515efb72c294f0f372c2e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for ergon-framework-python 0.2.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page