postflight

Turn-level failure detection for tool-calling agents

postflight reads agent traces you already emit and returns coded findings for failures that span a whole turn: a tool that declined three steps before the reply contradicting it, the same read issued eight times, a cache that never warmed. It calls no model and has no dependencies.

The failure is a relationship between steps. Scored one at a time, which is what an observation-scoped evaluator does, every step here passes.

The taxonomy

The codes are the stable interface. Filter on them, chart them, page on them. Renaming one is a breaking change.

Install

pip install postflight

No dependencies. Python 3.11+.

Use

The CLI is what a cron job or a CI step wants. Exit status is 1 when something faulted, 0 otherwise:

python -m postflight --langfuse --hours 24
python -m postflight --otel spans.jsonl

From Python:

from postflight import Config, faults, run_all
from postflight.adapters.langfuse import LangfuseAdapter, LangfuseClient

client = LangfuseClient(host, public_key, secret_key)
turns = LangfuseAdapter().turns(client.observations(hours=24))

for turn_id, findings in run_all(turns, Config()).items():
    for finding in faults(findings):
        print(turn_id, finding.code, finding.message)

run_all returns every finding, including Severity.INFO ones like GATE_FILTERED. Wrap it in faults() for anything a human reads first.

Output

Run against the fixture, a support agent built to fail this way and captured through OpenInference:

$ python -m postflight --otel tests/fixtures/openinference_support_turn.jsonl
  0x4069cd953e   TOOL_REFUSAL       1 tool call(s) declined in-body

1 turns, 1 flagged
  TOOL_REFUSAL       1

Not all detectors are live on this data:
  GATE_FILTERED: INERT - no quiet_kinds configured, so nothing is silent by design
  NO_CACHE_HIT: INERT - no generation reports cache usage, and unknown is not treated as zero

Each finding carries a detail dict, which is the part you act on:

{"calls": [{"tool": "send_notification",
            "result": "{'sent': False, 'reason': 'channel unavailable'}"}]}

The closing block comes from coverage(), and it is worth reading before the findings. A detector whose input is missing does not error, it just never fires, and an empty column looks the same as a clean agent. coverage() reports which detectors could not have fired on this data, so a zero elsewhere means something.

Expect SLOW_TURN and GATE_FILTERED to dominate any real window and expect the rare rows to carry the weight. Sort by severity, not by count.

The Turn contract

Detectors never see your trace format. They read Turn, so an adapter is a function from whatever you have to an ordered sequence of steps:

from postflight.model import Generation, ToolCall, Turn

Turn(
    id="…",
    # the surface: which agent, which channel
    kind="chat.turn",
    # ordered, because the sequence is the signal
    steps=(
        Generation(text="", input_tokens=900, model="…"),
        ToolCall(name="search", result={"count": 0}),
        Generation(text="I couldn't find it.", input_tokens=1200, model="…"),
    ),
)

A turn is a sequence, not a tree. Flattening is what lets a detector see a tool that declined several steps before the reply contradicting it.

Adapters ship for Langfuse and for OpenTelemetry / OpenInference. Copy postflight/adapters/otel.py, which handles flattened attributes, a span tree, and two timestamp encodings, so most of the awkward cases are already worked out. Read writing an adapter first; its three notes are each a mistake already made once.

Configuring

Thresholds, claim vocabulary, which surfaces owe a reply, and what each detector needs in order to fire at all: docs/configuring.md.

Status

Alpha. Detector codes are the public interface; thresholds, default vocabularies, and added detectors are not.

SemVer, with the 0.x convention the spec leaves undefined made explicit: while the major is 0, a minor bump may break the API and a patch may not. Release notes are on the Releases page.

Issues and PRs welcome. No response SLA.

Apache 2.0. See LICENSE.