stapel-alerts

An alert store for a fleet that has no Sentry, and the same capture() call whether one is connected or not. One row per BUG (fingerprint, status, count, first/last seen, the release that claimed the fix, whether it came back) rather than a search over log text — because the question a fix wave is reconciled against is 'which bugs are open, which are fixed, and which regressed', and that needs a status column. Every input is wired by installing the app: a WARNING+ log handler, the DRF 5xx hook, Celery task failures, core's DLQ parks, and the comm handler exceptions that are RETURNED rather than raised and which no except clause anywhere would otherwise see. Monoliths write in process; microservices POST with a per-service key, buffered and retried, and fall through to Telegram when the owner is unreachable — never over the bus, because the bus is the thing whose failures this records. A Prometheus blind-spot watchdog files the monitoring stack's own gaps as issues, on the principle that 'no alerts firing' is the observable state of both a healthy fleet and a blind one.

Part of the Stapel framework — composable Django apps that deploy as a monolith or as microservices without changing module code.

Install

Not published on PyPI yet. Install from source:

pip install git+https://github.com/usestapel/stapel-alerts

At a glance

Documentation

Errors: English · Español · Русский · OpenAPI · capabilities.json · llms.txt (for agents)

Why it exists

A day spent reading a production incident out of docker logs is the reason this exists. The signals were all there — a foreign key violation repeating on three accounts, twenty-two events parked in a dead-letter queue, a JWT refusal that turned out to be correct behaviour with a misleading log line — and every one of them had to be found by a human grepping containers, because nothing collected them.

This library collects them. Same interface with Sentry or without it:

from stapel_alerts import capture

capture(exc)
capture("STT provider exhausted", level="error", context={"provider": "x"})

Two-line mount

A monolith — this process owns the store:

INSTALLED_APPS = [..., "stapel_alerts"]
urlpatterns = [path("alerts/", include("stapel_alerts.urls"))]

That is all. STAPEL_ALERTS["SERVICE"] names the service in the tracker, and a system check tells you at boot if you forgot it.

A microservice — this process reports to the owner:

INSTALLED_APPS = [..., "stapel_alerts"]          # no urlconf mount
STAPEL_ALERTS = {
    "SERVICE": "svc-billing",
    "OWNER_URL": "https://api.example.com",
    "SERVICE_KEY": "<from `manage.py alerts_service svc-billing` on the owner>",
    "FALLBACK": {"TELEGRAM_BOT_TOKEN": "...", "TELEGRAM_CHAT_ID": "..."},
}

Nothing else changes. Every input below is wired by AppConfig.ready().

What it captures, without you instrumenting anything

Two models, because they answer two questions

Issue is the tracker: one row per bug, with status, count, first_seen/last_seen, and the version that fixed it. It is machine-addressable on purpose — an agent lists it, fixes the code, and closes the issue over the API.

ErrorEvent is one occurrence: the full trace, the context (redacted through core's redaction seam), the request path, the trace id.

Grouping is by similarity, not by exact match

Two tracebacks that differ only in a uuid, an id, a timestamp, a memory address or a checkout path are one issue. The normaliser strips those, the fingerprint is taken over the normalised top frame plus the escaping exception's message, and a near-duplicate scoring ≥ 0.9 over normalised frames joins the existing issue.

A short trace (fewer than four frames) groups only on an exact match after normalisation. Two four-line traces hit 0.9 by coincidence, and a wrong merge is worse than a duplicate issue: it hides a live bug behind one somebody has already marked fixed.

The rule is tested on real production traces, including the case that must not group — a different constraint on the same table, from the same frames.

Statuses, and where they come from

new ──▶ fixed ──▶ regressed
 │        ▲           │
 └── muted┘◀──────────┘

fixed is set by an API call or by CI on a commit referencing alerts:<issue-id>, with the version and sha that claim the fix. regressed is never set by a caller — the store sets it when a fixed issue receives a new event. A caller who could assert it could also decline to.

API

GET    /alerts/api/v1/issues              ?status= &level= &service= &since= &open= &offset= &limit=
GET    /alerts/api/v1/issues/{id}         + the last 20 events
PATCH  /alerts/api/v1/issues/{id}         {status, note, muted_until}
POST   /alerts/api/v1/issues/{id}/fix     {version, sha}
POST   /alerts/api/v1/report              {events: [...]}   X-Service-Key

The list is a page, {count, offset, limit, results} (IssuePage in the schema). limit is 1..200, default 50; an out-of-range value is clamped and the envelope echoes the limit that was applied. A PATCH carrying only muted_until is a mute (null = no deadline); leaving muted by any route clears the deadline.

Staff session for the tracker, service key for /report, and not the other way round: a reporter's key lives in every container in the fleet, so the blast radius of one leaking must not include everything the store has ever recorded. JSON only, stable uuid ids, ETag on the reads.

When the owner is unreachable

Buffer, retry with backoff, and after FALLBACK_AFTER_MINUTES send a digest to the NOTIFY seam — Telegram by default, through stapel-notifications' channel if that is configured, otherwise a direct Bot API call. A dead alert store must be loud, not silent. The buffer is bounded and drops oldest first: a reporter that ran out of memory holding alerts about an outage would be a second outage.

The monitoring watchdog

A Prometheus blind spot is itself an issue in this tracker, because "nothing is firing" is the observable state of a healthy fleet and of a monitoring stack that has stopped looking. Something outside the stack has to ask.

STAPEL_ALERTS = {
    "MONITORING": {
        "PROMETHEUS_URL": "http://prometheus:9090",
        "TARGETS": ["svc-billing", "svc-api"],     # up == 0, and no `up` at all
        "METRICS": ["alerts_open_total"],          # absent()
        "HEARTBEAT_ALERT": "Watchdog",             # a dead-man's switch
        "ALERTMANAGER_URL": "http://alertmanager:9093",
    },
}

# and the schedule, shipped by the library:
from stapel_alerts.beat import get_alerts_beat_schedule
CELERY_BEAT_SCHEDULE = {**get_alerts_beat_schedule(), ...}   # every 5 minutes

No Celery? manage.py alerts_watch_monitoring is the same pass, for cron. --dry-run prints what it would say without filing or notifying anything.

Each finding becomes a kind="monitoring" issue and a message on the NOTIFY seam, on every run that has one. That is deliberate: the notification thresholds (new / regressed / spike) would be silent on a blind spot's ninth consecutive run, which is not less urgent than its first. A check that stops failing closes its issue with fixed_in = recovered <ts>; one that comes back regresses it. A check that is no longer configured is not closed — nothing recovered, nobody looked.

Two facts on the comm bus

from stapel_core.comm import on_action

@on_action("alerts.issue.opened")
def page_somebody(event): ...

@on_action("alerts.issue.regressed")
def reopen_the_ticket(event): ...

Schemas are in schemas/emits/, registered by core's autoloader and validated on every emit. Only these two — an event per occurrence would put a failing loop's whole traffic on the bus and make every subscriber re-derive the grouping this module already did.

alerts.issue.regressed carries the release that claimed the fix next to the release that is running, which is the pair that separates "the fix is wrong" from "the fix is not deployed".

The emit happens after the issue has committed, in a transaction of its own — not in the ingest's atomic block, where the rest of the fleet emits. emit marks its transaction rollback-only when it fails, so emitting inside the ingest would mean a broken outbox deletes the alert. The bus is the thing whose failures this store records; it has to record them on the day the bus is what is broken.

Sentry

Set SENTRY_DSN (or STAPEL_ALERTS["SENTRY_DSN"]) and install the extra (pip install 'stapel-alerts[sentry]'): each event is forwarded and its Sentry id stored back on the row. Storing locally is never conditional on it.

Retention and GDPR

manage.py alerts_sweep drops events past their level's retention and closed issues with nothing left. An open issue is never swept, however old: deleting it would turn "unresolved" into "never happened". erase_subject (wired into the GDPR provider registry) drops the user_id link and scrubs context that repeats it, and keeps the failure — an alert store's rows are the record of a system failing, not a record about a person.

Metrics

alerts_new_total{level,service} (counter) and alerts_open_total{level,service} (gauge, declared at zero for every known pair, because a series that has never existed cannot be alerted on).

License

MIT — see LICENSE.

_{This page is assembled by stapel-readme from docs/readme.md plus the contract artifacts in docs/. Edit the prose in docs/readme.md; the badges, facts and links above and below it are generated — do not hand-edit README.md.}