EN | ES | RU

Avtomatika Worker SDK

Avtomatika Worker SDK is the official high-performance Python SDK for building distributed execution nodes (Workers / Holons) compatible with the Avtomatika orchestrator. It serves as an execution node (Shell) in a Holarchical Logic Network (HLN), coordinating with the orchestrator (Ghost) via the RXON (Reverse Axon Protocol).

The SDK automates all low-level communication concerns: task polling, dynamic registration, heartbeats, S3 payload management, Zero Trust authentication, and graceful shutdown.

🚀 Key Features

• Language: Python 3.11+
• Protocol: Native support for RXON (Reverse Axon Protocol) for HLN (Holarchical Logic Network).
• Communication Model:
  • PULL: Workers poll tasks from orchestrators (operates securely behind NAT/Firewall without public incoming ports).
  • WebSocket: Optional real-time bidirectional command channel (task cancellation, custom runtime commands).
• Zero Trust Security & Policy Enforcement:
  • Mandatory HMAC-SHA256 signing for all worker messages using WORKER_TOKEN.
  • Cryptographic verification of orchestrator_signature (sig) on incoming tasks via ORCHESTRATOR_SECRET_KEY.
  • Strict enforcement of allowed_skills task execution policies.
  • Automatic collection and reporting of task execution cost metrics (costs).
  • Identity Chain and Origin Worker ID support for provenance tracking across infinite holarchy layers.
  • Replay protection with timestamp validation.
• Traffic & Performance Optimization:
  • Telemetry Throttling (Heartbeat Deadband): Hardware telemetry (CPU/RAM/GPU) is only sent when a value changes by >5% or after a 60s forced interval, drastically saving network bandwidth.
  • ETag-Based Blob Caching: Heavy assets (e.g. AI model weights) are downloaded only once from S3, cached locally with ETag validation, and symlinked to task workspaces.
  • Async Results Uploader: Task results are queued via non-blocking asyncio.Queue with automatic retries and exponential backoff, instantly freeing the worker for the next task.
  • 3-Tier Skill Lifecycle: Supported (catalog), Available (dynamic limits), and Hot (cached in memory/VRAM).
  • Stable Hashing: Sends full skill catalog only when changed, using skills_hash for ultra-light heartbeats.
• S3 Streaming: High-performance payload streaming using obstore. Zero OOM on large files.
• AI-Agent Support: Supports Chain of Thought and Tool Use via OrchestratorClient dependency injection for subtask delegation.
• Hardware Awareness: Built-in monitoring for CPU, RAM, and NVIDIA GPUs (via psutil and GPUtil).
• Modern Observability:
  • Native distributed tracing and metrics with OpenTelemetry.
  • Automatic Trace Context Propagation: Extracts trace_id from tasks and propagates it to all events and child spans.
  • OTLP metrics push export via OTEL_EXPORTER_OTLP_ENDPOINT.

Table of Contents

• Core Concept: Holarchy, Ghost, and Shell
• Ecosystem
• Installation
• Quick Start
  • Usage as a Library
  • CLI Usage
• Configuration Reference
• Resilience & Connectivity
• Observability
• Security
• Contributing
• License

Core Concept: Holarchy, Ghost, and Shell

The Avtomatika architecture follows the principles of a Holarchical Logic Network (HLN):

• Orchestrator (The Ghost / Director): Manages high-level workflow state machines (Blueprints), tracks execution states, and coordinates jobs. It never initiates outbound network connections to workers.
• Worker (The Shell / Muscle): A specialized, autonomous execution unit (Holon). It connects to the Orchestrator from the bottom up via RXON, polls tasks, executes them in isolated environments, and returns results.
• Shell-Stacking (Matryoshka): Because workers and orchestrators adhere to standardized interfaces, an entire Orchestrator can be wrapped inside a Worker Shell. This enables the construction of recursive, self-similar fractal networks (Holarchies) of arbitrary depth.

Ecosystem

Avtomatika Worker SDK is an integral part of the Avtomatika distributed ecosystem:

• Avtomatika Orchestrator: High-performance state-machine workflow engine coordinating tasks and distributed blueprints.
• RXON Protocol: Shared lightweight reverse-connection protocol powering inter-node communication across the HLN ecosystem.
• HLN (Holarchical Logic Network): Architectural pattern, manifesto, and design specification for self-similar holarchies.
• Avtomatika Worker SDK: The official Python SDK for building execution workers (this repository).
• Full Example: Reference project demonstrating the orchestrator and workers in action.

Installation

• Install the core SDK only:

  pip install avtomatika-worker
  

• Install with S3 payload offloading support:

  pip install "avtomatika-worker[s3]"
  

• Install with Pydantic v2 schema inference support:

  pip install "avtomatika-worker[pydantic]"
  

• Install with OpenTelemetry metrics and tracing:

  pip install "avtomatika-worker[metrics]"
  

• Install all production features (Recommended):

  pip install "avtomatika-worker[s3,pydantic,metrics]"
  

• Install for development:

  pip install -e ".[dev]"
  

Quick Start

Usage as a Library

from avtomatika_worker import OrchestratorClient, TaskFiles, Worker

worker = Worker()


@worker.skill("hello_world")
async def my_skill(params: dict, files: TaskFiles):
    """Simple skill returning a greeting."""
    return {"message": f"Hello, {params.get('name', 'World')}!"}


@worker.skill("ai_agent_reasoning")
async def agent_skill(params: dict, orchestrator_client: OrchestratorClient):
    """AI agent skill delegating a subtask (tool use) via OrchestratorClient."""
    search_result = await orchestrator_client.call_skill("web_search", {"query": params["search_query"]})
    return {"result": f"Based on web search: {search_result['data']}"}


@worker.on_command("reboot")
async def handle_reboot(command: dict):
    """Custom command received over the real-time WebSocket channel."""
    print("Rebooting worker...")


if __name__ == "__main__":
    worker.run()

CLI Usage

The SDK includes a built-in CLI tool for managing worker lifecycles, health probes, and live-reload during development:

# Run the worker with health-check server on port 8083 (default)
worker run --app my_worker:worker

# Run in development mode with automatic reload on code changes
worker run --app my_worker:worker --reload

# Run on a custom port without the health-check server
worker run --app my_worker:worker --port 9000 --no-health-check

Configuration Reference

All settings can be configured via environment variables:

Resilience & Connectivity

• Independent Orchestrator Managers: Each orchestrator connection is handled by an isolated background task. Outages or rate limits on one orchestrator never block others.
• Smart Backoff & Retry Storm Protection: Unified exponential backoff honors the HTTP Retry-After header (seconds or HTTP date). Errors with status 429 without Retry-After enforce a mandatory 30-second safety floor to prevent retry storms.
• Heartbeat Debouncing: Throttles heartbeats to a maximum of once every 2 seconds. State updates during the cooldown are coalesced and dispatched cleanly.
• Graceful Shutdown: Handles SIGINT and SIGTERM cleanly, awaiting active task completion and notifying orchestrators prior to process exit.

Observability

Avtomatika Worker SDK provides first-class observability using OpenTelemetry:

• Distributed Tracing: Workers extract trace_id from tasks and create child spans (task.{type}). All emitted events and S3 operations share the same trace.
• Push Metrics (OTLP): When OTEL_EXPORTER_OTLP_ENDPOINT is configured, metrics are automatically exported via OTLP.
• ObservabilityManager Injection: Handlers can accept obs: ObservabilityManager to create custom sub-spans and attach diagnostic metadata.

Security

Security is foundational to the Avtomatika ecosystem:

• See the full RXON / HLN Security Model.
• See the Avtomatika Security Policy.
• Detailed worker security policies are documented in SECURITY.md.

Contributing

We welcome contributions! Please review our Contributor Guide and Development Guide for environment setup and testing instructions.

License

This project is licensed under the Mozilla Public License 2.0 (MPL 2.0).