deliverability-guard

A sending circuit breaker for outbound email — it watches reputation, bounce and complaint signals per mailbox and throttles or pauses before a domain burns, and it refuses to trip on statistically meaningless data. Run deliverability-guard run to watch continuously, or check on a cron schedule for the same evaluation one-shot.

The honest limits

For senders under roughly 1,000 messages/day/provider, this is a leading-indicator and compliance monitor, not a statistically valid complaint-rate breaker. Overclaiming here would make it wrong in production.

That sentence is a credibility asset, not a hedge. It stays here, first, on purpose.

At cold-outbound volume — which is most of the volume this tool is built for — a single complaint moves the observed rate by amounts a fixed-window breaker cannot interpret honestly. This tool models that uncertainty explicitly instead of pretending it isn't there. See docs/statistics.md for the full argument and docs/limits.md for what the underlying data itself can't tell you, no matter how the statistics are done.

The 0.15-of-a-message problem

A sender running 50 emails/day through one mailbox — an entirely ordinary cold-outbound volume. Gmail's hard ceiling is a 0.3% spam rate.

0.3% × 50 = 0.15 messages

You cannot receive 0.15 of a complaint. The only things that can actually happen in a day are 0 complaints (0%) or 1 complaint (2% — 6.7× over the ceiling). There is nothing representable in between. A breaker that watches the raw percentage will fire on that single complaint, because 2% really is greater than 0.3% — and it will be wrong to do so, because one data point at this volume carries almost no information about the mailbox's true underlying rate.

deliverability-guard never looks at the raw percentage. It models the unknown true rate as a Bayesian posterior and trips on the lower bound of a 95% credible interval, not the point estimate — the same one complaint in 50 sends that reads as "2%, over the limit, pause it" to a naive tool reads as "nowhere near enough evidence" here. See docs/statistics.md for the full worked example, and run it yourself:

uv run python examples/demo.py

Quick start (dry-run)

Dry-run is the default everywhere in this project. It is not a separate code path — it's the exact same evaluation logic, with a no-op decorator standing in for the real provider call, so what you see in dry-run is what would actually happen. That identity is per-evaluation, not per-process: a dry-run run daemon's in-memory state DOES keep remembering a mailbox it already decided to pause, the same way a live daemon's would — so its second and later ticks correctly report "already paused," not "would pause" again. A dry-run check invoked from cron, by contrast, never accumulates that memory across separate invocations, because the decision log deliberately never persists a dry-run action as real pause history (a dry-run action never touched the real provider, so a restart must not read it back as one) — so cron reports "would pause" every single time. Neither ever makes a real provider call; only how many times each shape reports the same hypothetical decision differs.

git clone https://github.com/katekruger/deliverability-guard
cd deliverability-guard
uv sync
cp .env.example .env                            # fill in your provider API key
cp config/thresholds.example.yml config/thresholds.yml
uv run deliverability-guard check                # evaluate every mailbox once, print verdicts

check is the single-shot form of the fast loop — the smallest thing you can put in cron to get a running system, not just a library. It reads config/thresholds.yml, pulls each mailbox's stats from the provider named there, evaluates every one through the exact same engine.breaker.evaluate shown below, appends a decision record per mailbox to the decision log, and exits non-zero if any mailbox's verdict isn't OK — so a cron job's failure alerting just works.

run is the always-on form: the same evaluation, on a loop, forever (Ctrl-C to stop).

uv run deliverability-guard run                  # watches continuously until stopped

Every fast_interval_seconds (default 300 = 5 minutes) it re-pulls stats and re-evaluates every mailbox, exactly like check — they share one code path (loops.fast.evaluate_all_mailboxes), so the two can't drift apart. Every slow_interval_seconds (default 86400 = 24h) it looks at its own recent evidence and tightens the ladder if the last several evaluations have been creeping toward warn without crossing it — the same "your thresholds were too loose for the last few days" logic BUILD-PLAN.md §5 describes, sourced from the daemon's own history rather than a live Postmaster feed. See ADR 0004 for exactly what this does and doesn't implement yet — most importantly, this polls on an interval rather than receiving pushed webhooks.

status <mailbox> prints a mailbox's current breaker state, and resume <mailbox> is the only way a paused mailbox becomes active again (see ADR 0003 — there is no automatic path back from PAUSED, whether it was check or run that paused it).

Exit codes: see deliverability-guard --help's own exit-code section — that epilog (cli.py's _EXIT_CODE_EPILOG) is this project's one canonical copy of the map, and cli.py's module docstring points at the same place rather than keeping a second copy of its own. (CLOSE10-1: this line used to restate the map itself and drifted stale; CLOSE11-2: "restate it in one FEWER places" was still two places, so this is now a pointer, not a copy, per AGENTS.md item 5 — a test in tests/test_cli.py fails if this line ever grows the map's own detail back in.)

Selectable providers: instantly (needs INSTANTLY_API_KEY), smartlead (needs SMARTLEAD_API_KEY and SMARTLEAD_CAMPAIGN_ID — its stats endpoint is per-campaign, not global), lemlist (needs LEMLIST_API_KEY and LEMLIST_CAMPAIGN_ID, also per-campaign), apollo (needs APOLLO_API_KEY and APOLLO_CAMPAIGN_ID, also per-campaign), ses (needs SES_CONFIGURATION_SET_NAME; authenticates via boto3's normal AWS credential chain rather than a provider API key — AWS_REGION is read explicitly rather than relying on ambient AWS config), and noop, which needs no credentials at all and reports a small synthetic fixture — set provider: noop in config/thresholds.yml to exercise check/run end to end (config loading, aggregation, evaluation, the decision log, exit codes) without a live account.

uv run deliverability-guard status sender@yourdomain.com
uv run deliverability-guard resume sender@yourdomain.com   # only after a human has looked

To call the engine directly instead of through the CLI:

from datetime import UTC, datetime

from deliverability_guard.engine.breaker import DEFAULT_LADDER, BreakerStateStore, evaluate
from deliverability_guard.engine.posterior import DEFAULT_PRIOR
from deliverability_guard.providers.base import MailboxRef
from deliverability_guard.providers.instantly import InstantlyDriver

driver = InstantlyDriver(api_key="...")  # never contacted while dry_run=True
mailbox = MailboxRef(provider="instantly", mailbox_id="sender@yourdomain.com")

result = evaluate(
    driver=driver,
    mailbox=mailbox,
    sends=50,
    complaints=1,
    prior=DEFAULT_PRIOR,
    thresholds=DEFAULT_LADDER,
    state_store=BreakerStateStore(),
    dry_run=True,  # no code path can pause or throttle without dry_run=False, set on purpose
    now=datetime.now(UTC),
)

print(result.verdict)  # Verdict.OK -- one complaint in 50 sends isn't evidence of anything yet

The full public surface is the CLI (cli.py) plus the engine, providers, signals, and identity modules directly; examples/demo.py is the fastest way to see the engine run without any provider credentials at all.

Provider capability matrix

Capability declaration is the whole design of the provider driver interface (providers/base.py): of nine sequencer platforms surveyed, two cannot pause anything at all. An interface that assumed every provider could pause would already be wrong for close to half of what was surveyed. Every driver's pause()/throttle() is always callable and returns an explicit "unsupported" result rather than silently doing nothing when a provider lacks a capability.

Instantly is the reference implementation because it's the only surveyed vendor exposing both per-mailbox daily bounce data and per-mailbox pause — the minimum viable substrate for a circuit breaker. Smartlead's per-mailbox daily-limit endpoint is, in this project's view, the most underrated endpoint in the space: it's the difference between a circuit breaker and a kill switch, which is why it's the second driver built rather than a third pause-only one.

The "campaign only" pause marks above describe driver-API surface the breaker itself does not reach (CLOSE4-3). engine.breaker.evaluate's ladder acts strictly per-mailbox — it only ever constructs a MailboxRef, never a CampaignRef — so through the CLI today, PAUSE actually executes against exactly one provider (instantly), and THROTTLE against exactly one (smartlead); no single provider can do both. Lemlist, Apollo, and SES's campaign-/configuration-set-level pause() calls are real, tested driver methods (pause_campaign, abort_sequence, pause_configuration_set), reachable by calling them directly, but the breaker's own evaluation loop returns an honest UNSUPPORTED for those providers rather than ever constructing the CampaignRef that would let them act — deliberately: pausing an entire campaign to handle one mailbox's bad evidence is a disproportionate action this project has not decided the breaker should take automatically (see providers/smartlead.py's and providers/ses.py's own module docstrings for the same reasoning applied per-driver). Teaching the ladder to pause a CampaignRef when no per-mailbox primitive exists is real, undecided v0.2 scope, not a bug.

Thresholds

The default ladder (config/thresholds.example.yml, engine/breaker.py), keyed on the posterior's lower confidence bound, never the point estimate:

Sourced against:

Google's 0.30% and Amazon SES's 0.10% are NOT the same measurement, and this project never blends them into one number. Google's denominator is Gmail inbox-delivered, DKIM-authenticated mail to engaged users. SES's is mail to domains that return complaint feedback to SES. The ladder above picks threshold values informed by both, as provider-agnostic policy points — never as a claim that the two rates are directly comparable. 0.30% is a terminal threshold: by the time evidence is strong enough to be confident it's been crossed, the damage is already done, because complaint data itself lags 24h–3 days behind send. 0.10% is the natural amber line instead — simultaneously Google's own recommended target and SES's own review trigger, two independent providers converging on the same number as "you should be worried now." See engine/breaker.py's module docstring for the full rationale.

What this cannot see

• IP reputation. For the dominant cold-outbound topology (Google Workspace / Microsoft 365 shared sending pools), IP reputation is both unobservable to a sender and not theirs to fix. Google itself removed the DOMAIN_REPUTATION and IP_REPUTATION metrics from Postmaster Tools v2 — there is no metric to read even if this project wanted one. This isn't a gap to be filled later; it's a design constraint, stated in BUILD-PLAN.md.
• Inbox placement. This tool has no visibility into whether a delivered message actually landed in the primary inbox versus a spam or promotions folder — it can see bounces, complaints, and compliance verdicts, not placement.
• Postmaster's own confidence bounds — removed in v2. Google's v1 Postmaster API exposed userReportedSpamRatioLowerBound/UpperBound alongside the point estimate. v2 dropped them entirely (confirmed against the live discovery document, docs/postmaster-verdicts.md) — a real regression this project doesn't have a way to work around, only to be honest about: every Postmaster-sourced rate is treated as a bare point estimate, run through the same posterior machinery as any other rate.
• Which campaign caused a spike, without the identity scheme adopted first. Postmaster reports a domain-day scalar; the sequencer reports per-message events; there is no join key between them by default. identity/feedback_id.py and identity/subdomain_advisor.py fix this — but only for mail sent after they're adopted, and only if the underlying sending architecture actually follows the scheme. See docs/limits.md.
• Warmup adherence. Not implemented in v0.1's shipped surface — a heuristic comparison against a folklore ramp curve exists at experimental/warmup_advisor.py, quarantined there because nothing calls it yet, not because the logic is unfinished. When it ships for real, it will ship as explicitly-labeled vendor-consensus heuristics, not a standard — there is no RFC or independent research behind published warmup curves, and presenting folklore as authoritative would be this project's fastest way to lose credibility.

License

MIT