Pyknic

Pyknic is a modular Python application framework and toolkit designed for background task scheduling, distributed operations, secure RPC command execution, and unified storage/backup management.

It includes:

• pyknic-server — an event-driven server daemon managing multithreaded task pipelines, schedulers, and FastAPI-based services.
• bellboy — a feature-rich CLI utility and client for server interaction, remote command execution, secure backups, and data replication.
• pyknic.lib — a comprehensive library of reusable modules for asynchronous I/O, unified storage access (Local, SFTP, AWS S3), cryptography, scheduling, and authentication.

Table of Contents

• Key Features
• Architecture Overview
• Installation
• Configuration
• Quick Start
  • 1. Running the Server (pyknic-server)
    • Reverse Proxy (Required for Production)
  • 2. Using the CLI Client (bellboy)
• Core Library Modules
• Development and Testing
  • Running Unit Tests
  • Local CI with Docker Compose
  • Linting and Type Checking
• License

Key Features

⏱️ Task Scheduler & Execution Engine

• Flexible Scheduling: Execute tasks immediately, sequentially in chains (ChainedTasksSource), or periodically (CronTasksSource).
• Thread Management: Thread-safe task execution using ThreadedTask, ThreadExecutor, and execution priority queues.

🌐 FastAPI Server & Lobby RPC

• Integrated Server: Built on top of FastAPI and Uvicorn with dynamic sub-application registration.
• Lobby RPC API: Secure command dispatch system supporting arbitrary registered operations with typed validation.
• Asymmetric Security: RSA-signed JSON Web Tokens (JWT) with configurable audience, leeway, TTL, and response signing.
• Pluggable AAA: Multiple authentication providers including htpasswd (Argon2, BCrypt, SHA-512) and static bearer tokens.

🧰 bellboy CLI Tool

• Server Control: Execute server-side commands such as ping and resources remotely.
• Session Management: Authenticate using login (basic, token, trust) and store credentials in secure secret backends (shm or OS keyring).
• Flexible Formatting: Output results in human-friendly Rich terminal tables or structured JSON (-f json).

📦 Unified Storage & Backup System

• Unified Storage Interface (IOClientProto): Consistent API for interacting with Local, SFTP (Paramiko), S3 (Boto3), and VirtualDir storage targets.
• Bandwidth Throttling: Asynchronous read/write throttling with IOThrottler.
• Streaming Compression: Native compression support for gzip, bz2, lzma, and zstandard.
• archive_v1 Backup Format: Chunked, encrypted, and compressed backup archives created from files or command outputs directly to remote destinations.

🔐 Cryptographic Primitives

• RSA keypair generation, serialization, signature creation, and verification.
• Symmetric encryption (AES) and Key Derivation Functions (Argon2, PBKDF2, scrypt).
• Apache htpasswd parser with constant-time hash comparisons.

Architecture Overview

                      +---------------------------------------+
                      |               bellboy                 |
                      |    (CLI Client / Standalone Tool)     |
                      +-------------------+-------------------+
                                          |
                              REST / JSON | RSA-Signed JWT
                                          v
+---------------------------------------------------------------------------------+
|                                 pyknic-server                                   |
|                                                                                 |
|  +--------------------+     +---------------------+     +--------------------+  |
|  |     Scheduler      | <-> |       Datalog       | <-> |   FastAPI Server   |  |
|  |  (Cron / Chained)  |     |  (State & Signals)  |     |   (Uvicorn/Lobby)  |  |
|  +--------------------+     +---------------------+     +--------------------+  |
|                                                                                 |
+---------------------------------------------------------------------------------+

Installation

Prerequisites

• Python 3.11, 3.12, 3.13, or 3.14

Install from Source

Clone the repository and install the package:

git clone https://github.com/a1ezzz/pyknic.git
cd pyknic

# Install base package
pip install .

# Or install in editable mode with development & testing dependencies
pip install -e ".[all]"

Available extra requirement targets:

• dev: Development tools (twine, Babel, watchdog).
• test: Testing and linting suite (pytest, pytest-cov, flake8, mypy, type stubs).
• all: Combines both dev and test.

Configuration

Pyknic uses YAML configuration files supporting environment variable expansions (e.g. ${VAR:-default}).

Default application settings are loaded from package templates and can be merged with external configuration files or directories:

# config.yaml example
pyknic:
  apps:
    - "fastapi-init"       # Enable FastAPI server task

  fastapi:
    uvicorn_host: 0.0.0.0
    uvicorn_port: 8000
    swagger: yes           # Enable Swagger UI (/docs)
    apps:
      - "lobby"            # Enable Lobby RPC endpoint

    lobby:
      main_url_path: /api/v1/lobby
      private_key_default_size: 4096
      jwt:
        algorithm: "RS256"
        audience: "pyknic::lobby::user"
        ttl: 1800
      aaa_policies:
        admin_policy:
          provider: "htpasswd"
          file: "/etc/pyknic/htpasswd"
          allowed_commands:
            - "ping"
            - "resources"
            - "backup"

Configuration files can be specified via CLI arguments (-c, -C) or environment variables:

• PYKNIC_APP_FILE_CONFIG: Path to a single YAML configuration file.
• PYKNIC_APP_DIR_CONFIG: Path to a directory containing configuration files to merge.

Quick Start

1. Running the Server (pyknic-server)

Start the pyknic-server daemon with your custom configuration:

# Basic run with info-level logging (-vv)
pyknic-server -c config.yaml -vv

# Verbosity flags:
# -v: WARNING
# -vv: INFO
# -vvv: DEBUG

When started, the daemon boots up the scheduler, registers configured background tasks, and launches the FastAPI HTTP server.

Reverse Proxy (Required for Production)

The built-in HTTP server is intended to run behind a reverse proxy such as nginx. Do not expose pyknic-server directly to the network. The reverse proxy must:

• Terminate TLS: serve all client traffic over HTTPS and forward requests to the backend over a trusted network. pyknic-server itself does not provide TLS.
• Limit the HTTP request body size (e.g. via nginx client_max_body_size) to reject oversized Content-Length requests before they reach the application and exhaust server resources.

Example nginx configuration:

server {
    listen 443 ssl;
    server_name pyknic.example.com;

    ssl_certificate     /etc/nginx/ssl/pyknic.crt;
    ssl_certificate_key /etc/nginx/ssl/pyknic.key;

    # Enforce a maximum request body size
    client_max_body_size 1m;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

2. Using the CLI Client (bellboy)

The bellboy utility provides CLI commands to manage sessions, interact with remote servers, and perform operations.

Authenticate & Manage Sessions

# Login via HTTP Basic / htpasswd
bellboy login \
  --server.lobby-url http://127.0.0.1:8000/api/v1/lobby \
  --server.secret-backend shm \
  --authentication basic \
  --login admin \
  --secret.direct "my-secret-password"

# List active stored credentials
bellboy list-logins --secret-backend shm

# Discard active session
bellboy logout \
  --server.lobby-url http://127.0.0.1:8000/api/v1/lobby \
  --server.secret-backend shm

Ping and Server Diagnostics

# Ping the remote lobby server
bellboy ping --server.lobby-url http://127.0.0.1:8000/api/v1/lobby

# Inspect server resource consumption (CPU, memory, threads)
bellboy resources --server.lobby-url http://127.0.0.1:8000/api/v1/lobby

# Output in JSON format
bellboy -f json resources --server.lobby-url http://127.0.0.1:8000/api/v1/lobby

File Synchronization & Replication (copier)

Copy files across local filesystems, SFTP, and AWS S3 with throttling support:

# Copy from local filesystem to remote SFTP with 1 MB/s throttling
bellboy copier \
  --source 'file:///var/data/archive.tar.gz' \
  --destination sftp://backupuser@192.168.1.50/var/backups/archive.tar.gz \
  --throttling 1048576

# Copy from S3 bucket to local directory
bellboy copier \
  --source s3://my-bucket/database.dump \
  --destination 'file:///local/backups/database.dump'

Secure Encrypted Backups (backup)

Create compressed and encrypted backups (archive_v1 format) directly to remote targets:

# Backup directory tree to an S3 bucket with Zstandard compression
bellboy backup \
  --files /var/www/html /etc/nginx \
  --destination s3://my-backups/nginx-www.archive \
  --compression zstandard \
  --password "strong-passphrase"

Core Library Modules

Pyknic provides several standalone, modular packages under pyknic.lib:

Development and Testing

Running Unit Tests

Run the test suite with coverage reporting:

pytest

HTML coverage reports are generated automatically under docs/pytest/ and docs/pytest-coverage/.

Local CI with Docker Compose

Pyknic provides a self-contained local Concourse CI environment using Docker Compose:

# Run tests against Python 3.13 in isolated Concourse pipeline
PYTHON_VERSION="3.13" LOCAL_FILES="$(pwd)" docker compose -f docker/local-tests-compose.yaml run local-test

# Or run against Python 3.12
PYTHON_VERSION="3.12" LOCAL_FILES="$(pwd)" docker compose -f docker/local-tests-compose.yaml run local-test

While the stack is running, access the Concourse web UI at http://localhost:8080 (concourse / concourse).

To clean up containers and volumes:

docker compose -f docker/local-tests-compose.yaml down -v

See docker/README.md and concourse-ci/README.md for further CI documentation.

Linting and Type Checking

Enforce code formatting and type safety:

# Check PEP8 style compliance
flake8

# Run static type checks
mypy pyknic

License

This project is licensed under the GNU Lesser General Public License v3 - see the LICENSE file for details.