herr-ober 0.1.12

High-performance S3 ingress controller for Ceph RGW clusters using HAProxy and ExaBGP

Herr Ober

High-Performance S3 Ingress Controller (BGP/ECMP)

Herr Ober ("Head Waiter") is a lightweight, high-throughput (50GB/s+) ingress controller designed for Ceph RGW clusters. It uses HAProxy 3.3 (AWS-LC) for SSL offloading and ExaBGP for Layer 3 High Availability via ECMP.

Supported: Ubuntu, Debian, RHEL 10+ on Proxmox VMs (KVM)

---

Documentation

For deep internals, kernel tuning, and failure recovery logic, see architecture.md.

---

Quick Start

1. Proxmox VM Prerequisites

Before installing, ensure the VM is configured for 50GB/s throughput:

• CPU: Type host (AES-NI passthrough)
• Network: VirtIO with Multiqueue enabled (Queues = vCPUs)
• Hardware Watchdog: Add device Intel 6300ESB → Action: Reset

2. Install

One-liner (recommended):

curl -fsSL https://raw.githubusercontent.com/dirkpetersen/ober/main/install.sh | sudo bash
sudo ober bootstrap

Manual install:

sudo su -
apt install -y pipx
pipx ensurepath
source ~/.bashrc
pipx install herr-ober
ober bootstrap

3. Configure

Interactive wizard to set up BGP, VIPs, backends, and certificates.

sudo ober config

4. Verify

# Check prerequisites and configuration
ober doctor

# View service status
ober status

---

Usage

CLI Commands

ober bootstrap [path]     # Install and set up everything
ober config [--dry-run]   # Interactive configuration wizard
ober sync                 # Update external system whitelists
ober status               # Show current state (--json for scripting)
ober start|stop|restart   # Service management (stop gracefully withdraws BGP)
ober logs [-f] [-n N]     # View logs (--service http|bgp to filter)
ober doctor               # Diagnostic checks
ober test                 # Test BGP connectivity without starting services
ober upgrade              # Check and install updates
ober uninstall            # Clean removal

Updating Whitelists

Update external system whitelists with Slurm hostlists or IP addresses:

# Update all whitelists (interactive prompts)
ober sync

# Update specific whitelist
ober sync --routers "switch[01-04]"
ober sync --frontend-http "weka[001-100]"
ober sync --backend-http "rgw[01-08].internal"

Checking Health

# Full status with systemd service info
ober status

# JSON output for monitoring integration
ober status --json

# Direct health endpoint
curl http://127.0.0.1:8404/health

---

Failure & Recovery

Event	Recovery
Node Crash	Traffic fails over via ECMP (instant)
OS Freeze	Proxmox Watchdog hard-resets VM (10s)
HAProxy Crash	BGP withdraws immediately (BindsTo=)
Network Cut	BFD detects and tears down route (~150ms)

See architecture.md for detailed failure scenarios.

---

Development

# Clone and install dev dependencies
git clone https://github.com/dirkpetersen/ober.git
cd ober
pip install -e ".[dev]"

# Run tests
pytest

# Lint and format
ruff check .
ruff format .

# Type check
mypy ober/

---

License

MIT