splent_feature_mail

The mail feature of a SPLENT product line.

Transactional mail over SMTP, for every other feature: MailService, an outbox that records each mail with its state, its attempts and the last error, retries in the background through the framework's task runner, a screen for administrators with the mails that failed (/admin/mail), and splent feature:mail test to check the settings. Standard library only (smtplib, email), no third-party dependency.

A SPLENT feature is a Python package that a product composes with others. Its models, routes, services, templates and hooks are registered when the product starts. What this feature provides and requires is listed in the [tool.splent.contract] section of pyproject.toml, which splent feature:contract --write keeps in sync with the source.

Sending a mail

from splent_framework.services.service_locator import service_proxy

mail = service_proxy("MailService")
mail.send("ada@example.org", "Password changed", body)
mail.send(
    "ada@example.org",
    "Reset your password",
    body,
    html=html_body,
)

send(to, subject, body, reply_to=None, *, html=None, sensitive=None) returns True when the mail went or is on its way, False when it could not go. It raises nothing for a transport error, which is logged, recorded and shown on /admin/mail. It raises ValueError for a header with a line break and for an HTML part whose links are not the text's.

• to is an address or a list of them, body the plain-text part.
• html is the HTML part, sent as the text's alternative. It carries the same links as body, so a reader whose client shows one part only can follow every link; send refuses it otherwise, counting the links that differ and never quoting one. html_from_text(body) (also MailService.html_from_text) builds one from the text, escaped, with its paragraphs and its links, for a caller that has no HTML template yet.
• sensitive says whether a link in the mail carries a secret, a token that opens something. None, the default, takes any link to carry one; False says none does; True withholds the whole body, link or not (a one-time code).

The outbox

Every mail is written to mail_outbox before anything is sent, on a connection of its own and committed at once. It never waits for the caller's transaction and never commits what the caller left pending, so a mail is recorded whatever the caller does next, as it was sent whatever the caller did next when send talked to the server itself.

A row keeps the recipients, the sender, the subject, both parts, retryable, the attempts of its current round, last_error (the exception's type and message, never a traceback) and when it was created, last tried and sent.

No body that carries a token

A link in a reset or confirmation mail opens an account, and a copy of the outbox must open nothing. So what is kept depends on sensitive:

A mail that is not retryable is never sent again from the outbox, by the task runner, the sweep or an administrator, since that would mean keeping its token. Whoever waits for the link asks for a new one, which mints a new token: the reset form, confirmemail's resend (at most one every CONFIRM_EMAIL_RESEND_SECONDS, five minutes) and auth's /account/confirm-email. The alternative, keeping what the mail is rendered from and minting the token at send time, would make every caller hand over a renderer instead of a body, and build its links outside the request, where SERVER_NAME has to be set; it was left out.

Retries

A retryable mail goes to tasks.deliver through splent_framework.tasks, so TASKS_MODE decides where it runs: a thread of the web process (the default), an RQ worker (rq) or the caller (sync, the tests). A failure that may pass (a dropped connection, a timeout, a 4xx answer) is tried again after 1, 2, 4, 8 and 16 minutes, six attempts over about half an hour; a permanent one fails at once. An attempt claims its row (queued to sending in one UPDATE), so two attempts never send it both. When the runner cannot take the mail (rq with Redis down), one attempt is made in the caller, and the sweep sends it later if that fails too.

In thread mode a retry waits in the web process, and a restart loses it. The sweep, splent feature:mail retry, sends every mail queued and idle for 20 minutes and takes back an attempt that has not settled in 10 (its process died), one attempt each, in the command, whatever TASKS_MODE says. A process that dies after the server took a mail and before the row said so leaves it to the sweep, and the mail may arrive twice: delivery is at least once.

A database the migration has not reached yet has no outbox: send then sends at once, as before, and logs it.

For administrators

/admin/mail, in the Administration section of the menu (mail_outbox, order 90, with the number of failed mails as its badge), shows the mails that failed and the ones waiting for another attempt, with the reason. A failed retryable mail can be sent again, which starts a new round of attempts in the background, or discarded. One that carried a link is marked so and has no "Send again".

Commands

splent feature:mail test someone@example.org   # one mail now, and the server's answer
splent feature:mail retry                      # what lost its retries
splent feature:mail retry --failed             # ...and a new round for the failed
splent feature:mail purge                      # what is older than MAIL_OUTBOX_KEEP_DAYS

test prints where mail goes and how (never the password), sends a mail with a text and an HTML part through the outbox, tried once in the command, and exits 1 with the server's answer when it did not go.

retry and purge do not run by themselves. Schedule them on the host, as the user the app runs as: the sweep every 15 minutes, the purge once a day.

*/15 * * * * docker exec -u splent <product>_web_deploy splent feature:mail retry
20 4 * * *   docker exec -u splent <product>_web_deploy splent feature:mail purge

Both exit with 0 when there is nothing to do.

Configuration

[tool.splent.config_schema] types each one, all bound to the deployment, and splent_framework 1.22 resolves them when the app starts: a port that is not a number, a sender without an @ or a password in the product's pyproject.toml stops the start, naming the key. Then the feature refuses what no product means:

• MAIL_USE_TLS and MAIL_USE_SSL together;
• MAIL_TLS_VERIFY=false outside development;
• in production, a MAIL_USERNAME with neither TLS nor SSL, which would send the password in clear.

Production on the development defaults (localhost:1025) is logged.

TLS

Both TLS ways verify the server's certificate and its name with ssl.create_default_context(). smtplib without a context verifies neither, so whoever sat between the app and the server read the login and every reset link. MAIL_TLS_VERIFY=false skips the check only in development with splent_feature_mailhog installed; anywhere else the start refuses it, and so does every attempt. MailHog itself speaks plain SMTP, which is what splent_feature_mailhog sets (MAIL_HOST, MAIL_PORT, no TLS, no login).

Use it in a product

splent feature:add splent_feature_mail
splent db:upgrade

Develop

splent feature:test splent_feature_mail
splent feature:contract splent_feature_mail

Release

splent feature:release splent_feature_mail --dry-run --bump minor
splent feature:release splent_feature_mail

The first command rehearses the release and changes nothing. The second publishes it to the git forge and to PyPI, and a PyPI version is permanent.