compose-farm 0.8.0

Compose Farm - run docker compose commands across multiple hosts

Compose Farm

A minimal CLI tool to run Docker Compose commands across multiple hosts via SSH.

[!NOTE] Run docker compose commands across multiple hosts via SSH. One YAML maps services to hosts. Change the mapping, run up, and it auto-migrates. No Kubernetes, no Swarm, no magic.

• Why Compose Farm?
• Key Assumption: Shared Storage
• Limitations & Best Practices
  • What breaks when you move a service
  • Best practices
  • What Compose Farm doesn't do
• Installation
• Configuration
• Usage
  • Auto-Migration
• Traefik Multihost Ingress (File Provider)
• Requirements
• How It Works
• License

Why Compose Farm?

I run 100+ Docker Compose stacks on an LXC container that frequently runs out of memory. I needed a way to distribute services across multiple machines without the complexity of:

• Kubernetes: Overkill for my use case. I don't need pods, services, ingress controllers, or YAML manifests 10x the size of my compose files.
• Docker Swarm: Effectively in maintenance mode—no longer being invested in by Docker.

Compose Farm is intentionally simple: one YAML config mapping services to hosts, and a CLI that runs docker compose commands over SSH. That's it.

Key Assumption: Shared Storage

Compose Farm assumes all your compose files are accessible at the same path on all hosts. This is typically achieved via:

• NFS mount (e.g., /opt/compose mounted from a NAS)
• Synced folders (e.g., Syncthing, rsync)
• Shared filesystem (e.g., GlusterFS, Ceph)

# Example: NFS mount on all hosts
nas:/volume1/compose  →  /opt/compose (on nas01)
nas:/volume1/compose  →  /opt/compose (on nas02)
nas:/volume1/compose  →  /opt/compose (on nas03)

Compose Farm simply runs docker compose -f /opt/compose/{service}/docker-compose.yml on the appropriate host—it doesn't copy or sync files.

Limitations & Best Practices

Compose Farm moves containers between hosts but does not provide cross-host networking. Docker's internal DNS and networks don't span hosts.

What breaks when you move a service

• Docker DNS - http://redis:6379 won't resolve from another host
• Docker networks - Containers can't reach each other via network names
• Environment variables - DATABASE_URL=postgres://db:5432 stops working

Best practices

1. Keep dependent services together - If an app needs a database, redis, or worker, keep them in the same compose file on the same host

2. Only migrate standalone services - Services that don't talk to other containers (or only talk to external APIs) are safe to move

3. Expose ports for cross-host communication - If services must communicate across hosts, publish ports and use IP addresses instead of container names:

   # Instead of: DATABASE_URL=postgres://db:5432
   # Use:        DATABASE_URL=postgres://192.168.1.66:5432
   

   This includes Traefik routing—containers need published ports for the file-provider to reach them

What Compose Farm doesn't do

• No overlay networking (use Docker Swarm or Kubernetes for that)
• No service discovery across hosts
• No automatic dependency tracking between compose files

If you need containers on different hosts to communicate seamlessly, you need Docker Swarm, Kubernetes, or a service mesh—which adds the complexity Compose Farm is designed to avoid.

Installation

pip install compose-farm
# or
uv pip install compose-farm

Configuration

Create ~/.config/compose-farm/compose-farm.yaml (or ./compose-farm.yaml in your working directory):

compose_dir: /opt/compose  # Must be the same path on all hosts

hosts:
  nas01:
    address: 192.168.1.10
    user: docker
  nas02:
    address: 192.168.1.11
    # user defaults to current user
  local: localhost  # Run locally without SSH

services:
  plex: nas01
  jellyfin: nas02
  sonarr: nas01
  radarr: local  # Runs on the machine where you invoke compose-farm

Compose files are expected at {compose_dir}/{service}/compose.yaml (also supports compose.yml, docker-compose.yml, docker-compose.yaml).

Usage

# Start services (auto-migrates if host changed in config)
compose-farm up plex jellyfin
compose-farm up --all

# Stop services
compose-farm down plex

# Pull latest images
compose-farm pull --all

# Restart (down + up)
compose-farm restart plex

# Update (pull + down + up) - the end-to-end update command
compose-farm update --all

# Sync state with reality (discovers running services + captures image digests)
compose-farm sync              # updates state.yaml and dockerfarm-log.toml
compose-farm sync --dry-run    # preview without writing

# View logs
compose-farm logs plex
compose-farm logs -f plex  # follow

# Show status
compose-farm ps

Auto-Migration

When you change a service's host assignment in config and run up, Compose Farm automatically:

1. Runs down on the old host
2. Runs up -d on the new host
3. Updates state tracking

# Before: plex runs on nas01
services:
  plex: nas01

# After: change to nas02, then run `compose-farm up plex`
services:
  plex: nas02  # Compose Farm will migrate automatically

Traefik Multihost Ingress (File Provider)

If you run a single Traefik instance on one “front‑door” host and want it to route to Compose Farm services on other hosts, Compose Farm can generate a Traefik file‑provider fragment from your existing compose labels.

How it works

• Your docker-compose.yml remains the source of truth. Put normal traefik.* labels on the container you want exposed.
• Labels and port specs may use ${VAR} / ${VAR:-default}; Compose Farm resolves these using the stack’s .env file and your current environment, just like Docker Compose.
• Publish a host port for that container (via ports:). The generator prefers host‑published ports so Traefik can reach the service across hosts; if none are found, it warns and you’d need L3 reachability to container IPs.
• If a router label doesn’t specify traefik.http.routers.<name>.service and there’s only one Traefik service defined on that container, Compose Farm wires the router to it.
• compose-farm.yaml stays unchanged: just hosts and services: service → host.

Example docker-compose.yml pattern:

services:
  plex:
    ports: ["32400:32400"]
    labels:
      - traefik.enable=true
      - traefik.http.routers.plex.rule=Host(`plex.lab.mydomain.org`)
      - traefik.http.routers.plex.entrypoints=websecure
      - traefik.http.routers.plex.tls.certresolver=letsencrypt
      - traefik.http.services.plex.loadbalancer.server.port=32400

One‑time Traefik setup

Enable a file provider watching a directory (any path is fine; a common choice is on your shared/NFS mount):

providers:
  file:
    directory: /mnt/data/traefik/dynamic.d
    watch: true

Generate the fragment

compose-farm traefik-file --all --output /mnt/data/traefik/dynamic.d/compose-farm.yml

Re‑run this after changing Traefik labels, moving a service to another host, or changing published ports.

Auto-regeneration

To automatically regenerate the Traefik config after up, down, restart, or update, add traefik_file to your config:

compose_dir: /opt/compose
traefik_file: /opt/traefik/dynamic.d/compose-farm.yml  # auto-regenerate on up/down/restart/update
traefik_service: traefik  # skip services on same host (docker provider handles them)

hosts:
  # ...
services:
  traefik: nas01  # Traefik runs here
  plex: nas02     # Services on other hosts get file-provider entries
  # ...

The traefik_service option specifies which service runs Traefik. Services on the same host are skipped in the file-provider config since Traefik's docker provider handles them directly.

Now compose-farm up plex will update the Traefik config automatically—no separate traefik-file command needed.

Combining with existing config

If you already have a dynamic.yml with manual routes, middlewares, etc., move it into the directory and Traefik will merge all files:

mkdir -p /opt/traefik/dynamic.d
mv /opt/traefik/dynamic.yml /opt/traefik/dynamic.d/manual.yml
compose-farm traefik-file --all -o /opt/traefik/dynamic.d/compose-farm.yml

Update your Traefik config to use directory watching instead of a single file:

# Before
- --providers.file.filename=/dynamic.yml

# After
- --providers.file.directory=/dynamic.d
- --providers.file.watch=true

Requirements

• Python 3.11+
• SSH key-based authentication to your hosts (uses ssh-agent)
• Docker and Docker Compose installed on all target hosts
• Shared storage: All compose files at the same path on all hosts (NFS, Syncthing, etc.)

How It Works

1. You run compose-farm up plex
2. Compose Farm looks up which host runs plex (e.g., nas01)
3. It SSHs to nas01 (or runs locally if localhost)
4. It executes docker compose -f /opt/compose/plex/docker-compose.yml up -d
5. Output is streamed back with [plex] prefix

That's it. No orchestration, no service discovery, no magic.

License

MIT