fleet-framework 0.2.0

generic distributed-automation framework — master/worker, browser pool, anti-bot helpers, and abstract automation contracts (SERP, content, news, place, marketplace, jobs, social)

Fleet

A framework for distributed automation fleets. One master, many workers, typed output streams between them. Born out of a production CAPTCHA-solving farm, generalized to fit any "N machines each doing M parallel things" workload.

What it is

Two pip packages. fleet-core is the framework: master, worker shell, reconcile loop, Redis-backed store, output streams, event bus, a small dashboard. fleet-browser is an optional add-on that ships a battle-tested Chromium pool with fingerprint rotation, proxy auth, and orphan-process cleanup.

You write a Python class subclassing ContinuousAutomation or BatchAutomation and a Pydantic config. Ship it as a pip package with one entry-point line. Install it on the master and on each worker host and the framework wires up everything else: config push, validation, slot lifecycle, recycle, heal, stats, dashboard, stream-based inter-automation comms.

What's already in the box

• Master process: REST + WebSocket API, auto-discovered plugins, dashboard, Redis-backed state.
• Worker process: pulls config, runs N concurrent slots, reconciles to desired state, heals on failure.
• Reconcile loop: pure-functional diff, smallest-converging-plan semantics, slot recycle, self-heal.
• Typed output streams: ZSET-backed, bounded by length and TTL, one per automation type, pop or peek over REST or in-process.
• Event bus: Redis pubsub for lightweight signals.
• Auth: bearer tokens, admin vs worker, constant-time comparison. WebSocket handshake validated before accept.
• Hardening from the original farm: tini in the worker container reaps zombies, psutil-based process-tree cleanup catches orphans, recycle's heal step closes the "zero-slot lockout" failure mode.

What you write

Roughly 30 lines per automation. Here's a fully working example:

import asyncio
import httpx
from fleet.core import BaseConfig, ContinuousAutomation, register

class PingerConfig(BaseConfig):
    url: str
    interval_seconds: float = 5.0

@register("pinger")
class Pinger(ContinuousAutomation[PingerConfig]):
    Config = PingerConfig

    async def run_slot(self, ctx):
        async with httpx.AsyncClient(proxy=ctx.proxy) as client:
            while not ctx.shutdown.is_set():
                r = await client.get(ctx.config.url)
                await ctx.emit({"status": r.status_code})
                await asyncio.sleep(ctx.config.interval_seconds)

Plus one line in your pyproject.toml:

[project.entry-points."fleet.automations"]
pinger = "my_package:Pinger"

Quickstart

git clone <repo-url> fleet && cd fleet
python3 -m venv .venv && source .venv/bin/activate
pip install -e packages/fleet-core
pip install -e examples/hello-world

# in one terminal:
redis-server --port 6379 --appendonly yes
# in another:
FLEET_ADMIN_TOKEN=admin FLEET_WORKER_TOKEN=worker fleet master
# in another:
MASTER_URL=http://localhost:8000 FLEET_WORKER_TOKEN=worker fleet worker --type hello-world --id w1
# in another:
curl -X PATCH http://localhost:8000/api/v1/automations/hello-world/workers/w1/config \
  -H "Authorization: Bearer admin" -H "Content-Type: application/json" \
  -d '{"enabled": true, "slots": 2}'
curl -X POST http://localhost:8000/api/v1/automations/hello-world/output/pop \
  -H "Authorization: Bearer admin" -H "Content-Type: application/json" \
  -d '{"n": 5}'

Walkthrough with explanations: docs/getting-started/quickstart.md.

Repo layout

fleet/
├── packages/
│   ├── fleet-core/      master, worker, reconcile, store, streams, event bus, dashboard, CLI
│   └── fleet-browser/   optional Chromium pool, fingerprint, humanizer, proxy auth, cert
├── examples/
│   ├── hello-world/     minimal ContinuousAutomation (emit greetings on a timer)
│   └── echo-consumer/   consumes another automation's output stream
└── docs/                GitBook-format documentation (also lives on the `docs` branch)

Documentation

Full docs in docs/, rendered as a GitBook from the docs branch.

• Introduction
• Installation
• Quickstart
• Your first automation
• Architecture
• Writing a continuous automation
• Writing a batch automation
• Making two automations talk
• Browser-based automations
• Deployment
• REST API
• WebSocket protocol
• Authentication
• Troubleshooting

Origins

Fleet started as the control plane for a Cloudflare Turnstile-solving farm. That farm hit every interesting failure mode in distributed automation: zombie subprocess accumulation, master state wipe on Redis restart, gen-counter regression, slot-recycle drops with no heal, and config drift across heterogeneous workers. Each of those has a fix baked into the framework.

The framework is what the original farm should have been from day one. It's deliberately small. If you want a scheduler, an orchestrator, or a service mesh, this isn't it. If you want a thin layer that turns "N hosts doing repetitive work" into a manageable system, it is.

Status

Pre-alpha. The public API will change. No PyPI release yet — install from source. Production deployments are running on it but with a lot of manual operational knowledge to back them up. Watch the changelog before relying on anything sensitive.

See ROADMAP.md for the path to v0.1.

License

MIT. See LICENSE.