kuhl-haus-magpie 0.5.4

Site monitoring and synthetic transaction service

kuhl-haus-magpie

Site monitoring and synthetic transaction service built on Django and Celery.

Magpie runs scheduled canary checks against your HTTP endpoints and reports results to Graphite for dashboarding and alerting. It provides a Django Admin interface for managing endpoints and a REST API for programmatic access.

What It Does

Magpie performs three types of canary checks on configured endpoints:

Check	What It Does
HTTP Health Check	Sends a GET request and validates the response status code or JSON body
TLS Certificate Check	Connects via SSL and calculates days until certificate expiration
DNS Check	Queries configured DNS resolvers and validates the response

Results — including response times, error counts, and check-specific attributes — are posted to a Graphite server via the Carbon pickle protocol.

Architecture

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│ Celery Beat  │────>│ Celery Worker│────>│  Canary Task │
│ (scheduler)  │     │              │     │  Functions   │
└──────────────┘     └──────────────┘     └──────┬───────┘
                                                 │
                           ┌─────────────────────┼─────────────────────┐
                           │                     │                     │
                     ┌─────▼───────┐       ┌─────▼──────┐      ┌───────▼─────┐
                     │ HTTP Health │       │  TLS Check │      │  DNS Check  │
                     │   Check     │       │            │      │             │
                     └─────┬───────┘       └─────┬──────┘      └───────┬─────┘
                           │                     │                     │
                           └─────────────────────┼─────────────────────┘
                                                 │
                                          ┌──────▼──────┐     ┌──────────┐
                                          │ CarbonPoster│────>│ Graphite │
                                          │  (pickle)   │     │  Server  │
                                          └─────────────┘     └──────────┘

┌──────────────┐     ┌──────────────┐
│ Django Admin │────>│  PostgreSQL  │  (SQLite for development)
│  (Unfold)    │     │   Database   │
└──────────────┘     └──────────────┘

┌──────────────┐
│  Django Web  │──── /health, /healthz, /admin/, /api/
│  (Gunicorn)  │
└──────────────┘

Celery Beat triggers canary tasks on a schedule you define through the Django Admin. Celery Workers execute the checks and batch-post metrics to your Graphite server. The Django web application provides endpoint management, health check endpoints, and a Swagger/ReDoc API.

Tech Stack

Component	Technology
Language	Python ≥ 3.12
Web framework	Django 5.2
Task queue	Celery + django-celery-beat + django-celery-results
API	Django REST Framework + drf-yasg (Swagger/ReDoc)
Admin theme	django-unfold
Metrics transport	Graphite Carbon pickle protocol
DNS checks	dnspython
TLS checks	pyOpenSSL
Database	PostgreSQL (production) / SQLite (development)
ASGI server	Uvicorn + Gunicorn

Quick Start

Install from PyPI

pip install kuhl-haus-magpie

Run with Docker

Pre-built container images are published to GitHub Container Registry. A full deployment requires four containers:

Image	Purpose
ghcr.io/kuhl-haus/kuhl-haus-magpie:latest	Django web application
ghcr.io/kuhl-haus/kuhl-haus-magpie-worker:latest	Celery worker (executes canary tasks)
ghcr.io/kuhl-haus/kuhl-haus-magpie-beat:latest	Celery Beat (task scheduler)
ghcr.io/kuhl-haus/kuhl-haus-magpie-flower:latest	Flower (Celery monitoring dashboard)

Each container runs docker-entrypoint.sh on startup, which bootstraps the database (migrations, static files, superuser creation).

Development Setup

git clone https://github.com/kuhl-haus/kuhl-haus-magpie.git
cd kuhl-haus-magpie
pip install -r requirements.txt
pdm install

# Run the Django development server
python -m kuhl_haus.magpie.manage runserver 0.0.0.0:8000

# Run tests
pdm run pytest tests -v

# Run tests with coverage
pdm run pytest --cov=kuhl_haus.magpie --cov-report=html tests -v

Configuration

Magpie is configured entirely through environment variables. No configuration files are required.

Application Settings

Variable	Purpose	Default
DJANGO_SECRET_KEY	Django secret key	Insecure dev default
DJANGO_DEBUG	Enable debug mode	True
DJANGO_ALLOWED_HOSTS	Comma-separated allowed hosts	Derived from MAGPIE_DOMAIN
MAGPIE_DOMAIN	Domain for cookies, CSRF, and CORS	localhost
DISABLE_HTTPS	Disable HTTPS cookie/session settings	False

Database

Variable	Purpose	Default
POSTGRES_HOST	PostgreSQL host (if unset, uses SQLite)	None
POSTGRES_DB	Database name	magpie
POSTGRES_USER	Database user	magpie
POSTGRES_PASSWORD	Database password	magpie
POSTGRES_PORT	Database port	5432

Celery

Variable	Purpose	Default
CELERY_BROKER_URL	Message broker URL (Redis or RabbitMQ)	None

Metrics (Graphite)

Variable	Purpose	Default
CARBON_SERVER_IP	Graphite Carbon server IP	None
CARBON_PICKLE_PORT	Carbon pickle port	2004
NAMESPACE_ROOT	Root namespace for metrics	default
METRIC_NAMESPACE	Metric namespace	dev

Bootstrap

Variable	Purpose	Default
DJANGO_SUPERUSER_USERNAME	Initial admin username	admin
DJANGO_SUPERUSER_EMAIL	Initial admin email	[e-mail]
DJANGO_SUPERUSER_PASSWORD	Initial admin password (required for creation)	None

Container Metadata

Variable	Purpose	Default
IMAGE_VERSION	Image version (shown in /healthz)	Unknown
CONTAINER_IMAGE	Container image name (shown in /healthz)	Unknown
FLOWER_DOMAIN	Flower dashboard domain (shown on index page)	localhost:5555

Endpoints

Path	Method	Description
/	GET	Site map with links to admin, API docs, and Flower
/health	GET	Simple health check — returns OK (plain text)
/healthz	GET	Detailed health check — returns JSON with version info
/admin/	GET	Django Admin (Unfold theme)
/api/endpoints/	CRUD	Endpoint management API
/api/resolvers/	CRUD	DNS resolver management API
/api/resolver-lists/	CRUD	DNS resolver list management API
/api/scripts/	CRUD	Script configuration management API
/api/	GET	Swagger UI
/redoc/	GET	ReDoc API documentation

Metrics Format

Magpie posts metrics to Graphite in the following path structure:

{namespace}.{app_name}.mnemonic.{endpoint_mnemonic}.{metric_name}
{namespace}.{app_name}.hostname.{hostname}.{metric_name}

Tagged metrics (when metadata is configured):

{namespace}.{app_name}.mnemonic.{endpoint_mnemonic}.{metric_name};tag1=val1;tag2=val2

Counters

Every canary check records these counters:

Counter	Meaning
requests	Number of check attempts
responses	Successful responses
errors	Expected errors (bad status code, unhealthy response)
exceptions	Unexpected exceptions (timeouts, connection failures)

Check-Specific Attributes

HTTP Health Check: response_time, response_time_ms, status_code, text, request_time, request_time_ms, version

TLS Check: response_time, response_time_ms, days_until_expiration, expires_today, is_valid

DNS Check: response_time, response_time_ms, truncated, rcode, result

Managing Endpoints

Endpoints are managed through the Django Admin at /admin/. Each endpoint can be configured with:

• Mnemonic — unique identifier used in metric paths

• Hostname and port — target server

• Environment — used for metric namespace grouping (e.g., prod, staging, dev)

• Health check settings — scheme, path, expected status code, JSON response validation

• TLS check — enable/disable certificate monitoring

• DNS check — enable/disable with assigned resolver list

• Timeouts — configurable connect and read timeouts

• Ignore flag — temporarily disable an endpoint without deleting it

Canary task schedules are configured through Celery Beat in the Django Admin under Periodic Tasks.

Security

• See SECURITY.md for vulnerability reporting guidelines

• All releases are signed with Sigstore

• CodeQL scanning runs on every push

• Dependabot monitors pip and Docker dependencies

Contributing

See CONTRIBUTING.rst for development guidelines.

License

This project is licensed under the terms of the LICENSE.txt file.

Note

This project was set up using PyScaffold 4.6.