silentfail

Verify that your automation actually did the work, inside your own process.

A deadline catches a workflow that stops running. This catches one that runs, reports success, and does not do the work.

You describe the job in the dashboard, read the checks it compiles into, and approve them. From then on this package checks each run against those checks next to your own data. A check that compares values sends us the verdict and nothing else. The values it was computed from never leave your machine, and there is no field on the wire format that one would fit in.

One kind of check does transmit. A rule may include a judged check, for a question that comparing values cannot settle: whether a summary reflects the email it came from, whether a reply answers what was asked. A judged check names the fields it needs, you see that list before you approve the rule, and only the fields it names are sent, to a maximum of six. A rule with no judged check sends nothing but the verdict.

Python 3.9 or newer. No dependencies, deliberately: this installs into an environment that is already pinned to something.

Use

import os
from silentfail import Supervisor, Claim

supervisor = Supervisor.start(
    api_key=os.environ["SILENTFAIL_API_KEY"],
    monitor_id="cmu0480ko0007ssi0f1zz7d60",
)

# ... your job runs, and files invoice INV-4471 ...

supervisor.check(
    run_id=job_id,
    claim=Claim(status="success", reference="INV-4471"),
    evidence=lambda: ledger.find(invoice_no="INV-4471"),
)

supervisor.flush()   # for a Lambda, a cron job, anything that exits when done

claim is what your run says it did. evidence is what an independent source says happened, fetched by your code, from your systems.

The two rules

start() raises. check() never does.

start() runs once, at startup, with a developer watching. A mistyped key or a monitor with no approved rule is a deploy-time problem, and finding out at three in the morning by noticing that nothing has been verified for a month is not acceptable, so it fails immediately and says what to fix.

check() runs in production at the end of a real job. It has no raising path, for any input, any network condition, or anything that has gone wrong on our side. A monitoring tool that takes down the thing it monitors is worse than no monitoring tool.

An account problem is not a setup mistake. An unpaid invoice or a paused subscription is decided on our side and can happen months after this was deployed, so it logs loudly and your process carries on running unverified. Our billing system does not get to stop your production job.

It does not block

The rule is fetched once at startup and cached. Verdicts go out on a background daemon thread, so check() returns as soon as the local evaluation is done and a daemon thread never delays your interpreter exiting. Call flush() when you want to wait for them.

Your evidence callable is bounded by a timeout (10 seconds by default). Python cannot kill the thread still waiting on a slow query, so that thread is left to finish on its own, but your call returns at the deadline. Timing out is reported honestly as "could not check".

Three outcomes, not two

Every check is passed, failed, or could_not_check, and the third is the important one. A check that did not run is never reported as a pass.

Agreement with the Node package

The evaluator here is a reimplementation, not a binding. The two are held together by supervisor-conformance/, a suite of rule, input and expected verdict fixtures that both packages run, written from the format's documented semantics rather than recorded from either implementation.

Four Python behaviours would silently disagree with JavaScript without the fixtures catching them, and all four have cases:

• 0 == False and 1 == True are true here and false there. Equality compares the type first.
• isinstance(True, int) is true, so a boolean would satisfy every numeric comparison. Numeric tests exclude bool explicitly.
• JSON 1 parses to int and 1.0 to float, where JavaScript has one number type. Whole-number tests ask about the value, not the type.
• Dictionary lookup has no prototype chain, so unlike the JavaScript version there is nothing here that needs guarding against a path resolving to constructor.

Run them:

cd packages/silentfail-python && python -m unittest discover -s tests -t .