clerk

A Python library for managing civic data pipelines for civic.band. Clerk handles the complete workflow of fetching, processing, and deploying civic meeting data including minutes and agendas.

Features

• Distributed Task Queue: RQ-based distributed processing with parallel OCR and horizontal scaling
• Site Management: Create and manage civic sites with metadata
• Data Pipeline: Automated fetch → OCR → compilation → deploy workflow
• Plugin System: Extensible architecture using pluggy for custom fetchers and deployers
• Full-Text Search: Automatic FTS index generation for searchable meeting data
• Database Management: SQLite-based storage with per-site and aggregate databases
• Observability: Structured JSON logs on stdout shipped to VictoriaLogs via Vector, OTel traces to VictoriaTraces, Prometheus metrics per worker (see docs/observability.md)

Quick Install

pip install "civicband-clerk[pdf,extraction] @ git+https://github.com/civicband/clerk.git"

Documentation

Setup

• macOS Setup - Complete installation guide for macOS
• Linux Setup - Complete installation guide for Linux
• Single-Machine Workers - Configure workers on one machine
• Distributed Workers - Scale across multiple machines
• Verification - Test your setup

Operations

• Daily Tasks - Common operational tasks
• Monitoring - Health checks and metrics
• Troubleshooting - Fix common issues
• Scaling - Add workers and scale horizontally

Reference

• CLI Reference - Complete command-line reference
• Python API - Python library reference
• Plugin API - Plugin development guide

Guides

• Your First Site - Complete beginner tutorial
• Worker Architecture - Understanding task queues
• Custom Fetcher - Build a fetcher plugin
• Production Checklist - Pre-launch validation

Quick Start

# Create a new site
clerk etl new

# Update a site (enqueues fetch → OCR → compilation → deploy)
clerk etl update --subdomain example.civic.band

# Check status
clerk status

See Your First Site Tutorial for a complete walkthrough.

Architecture

Clerk uses a distributed task queue (RQ) with specialized worker types:

• fetch - Download meeting data from city websites
• ocr - Extract text from PDFs (parallel, CPU-intensive)
• compilation - Build databases and coordinate pipeline
• extraction - Entity and vote extraction (optional, memory-intensive)
• deploy - Upload to storage/CDN

Workers can run on a single machine or distributed across multiple machines for better performance.

See Worker Architecture Guide for details.

Contributing

See CONTRIBUTING.md for development setup and guidelines.

License

BSD 3-Clause License - See LICENSE for details.

Links

• Documentation: docs/
• Issues: https://github.com/civicband/clerk/issues
• civic.band: https://civic.band