pantheon-guardrails

Three guardrails for LLM agents, extracted from a private production system:

• ConstitutionScorer — scores a draft reply against a weighted rubric, but only when the draft looks like it's worth paying for.
• SafetyLayer — redacts emails, phone numbers and card-like digit runs, while allowing a business to publish its own contact details.
• CrisisProtocol — detects a user in distress and breaks the assistant's persona.

Apache-2.0. Python 3.11+. One runtime dependency (pydantic, and only for the rubric model).

pip install pantheon-guardrails

What is actually interesting here

Most of this is unremarkable. Two ideas are worth the read.

1. Conditional judging

Using an LLM to grade another LLM's output is a well-known technique and a well-known cost — you double your inference bill to check work that is usually fine.

So the judge doesn't always run. A cheap regex pre-pass decides whether a draft is worth checking, and it fires on the things a customer acts on: money, quoted times and dates, percentages, bookings, refunds, deletions, anything medical or legal.

scorer = ConstitutionScorer(rubric, judge=my_judge)

await scorer.score_if_needed("The garden is lovely this time of year.", manifest)
# -> None. No LLM call. Nothing here can hurt anybody.

await scorer.score_if_needed("I've booked you in for 3pm Tuesday, that's £40.", manifest)
# -> Score(total=0.87, guidance=None). A time, a date and a price: worth checking.

The heuristic is a first cut and is stated as such in the source. It has not been calibrated against a labelled set in this repo. Override _should_judge or pass always_judge=True if your risk profile differs.

2. The judge must be a different model

scorer = ConstitutionScorer(rubric, judge=judge)   # required, not optional

A generator asked to mark its own work shares its own blind spots — the same training, the same failure modes, the same confident wrongness. The judge is a required argument specifically so that "which model grades this?" is a decision you make rather than a default you inherit.

The Judge protocol is two methods wide, so adapting whatever client you already use takes about five lines:

class AnthropicJudge:
    def __init__(self, client, model="claude-haiku-4-5"):
        self.client, self.model = client, model

    async def complete(self, messages):
        r = await self.client.messages.create(
            model=self.model, max_tokens=512,
            system=next(m.content for m in messages if m.role == "system"),
            messages=[{"role": "user", "content": m.content}
                      for m in messages if m.role == "user"],
        )
        return type("C", (), {"text": r.content[0].text})()

3. A failed evaluation is not a passing one

score = await scorer.score_if_needed(draft, context)

if score is None:            # no judging was required
    ...
elif not score.evaluated:    # the judge ran and could not be trusted -> total 0.0
    ...

Until v0.2.0 this got it backwards. _finite01() clamped an out-of-range score up: a judge replying {"empathy": 99} — a plausible way for a model to answer "score this out of 100" — produced a perfect 1.0 and passed any threshold. And an unparseable reply produced an empty score map, whose missing principles each defaulted to a neutral 0.5, ranking a broken evaluation above an honest zero.

Both now return Score(total=0.0, evaluated=False). The flag exists because None already meant "no judging was needed", and overloading it would have made a broken judge indistinguishable from a skipped one — in the first version of the fix it did exactly that, and a downstream quality gate treated the failure as a pass. A sentinel that gains a second meaning silently changes every call site that reads it.

The defect was found by an external reviewer reading the published source, reproduced with a scripted judge, and the regression tests are mutation-checked: re-clamping upward, restoring the 0.5 default, or pinning evaluated=True each turn a test red.

4. A failed evaluation has a deadline, and says which principle failed

scorer = ConstitutionScorer(rubric, judge=judge, timeout_s=20.0)   # 20s is the default
score = await scorer.score_if_needed(draft, manifest)

if score is None:
    ...                       # no judging was required (low-stakes draft, or threshold 0)
elif not score.evaluated:
    ...                       # the judge timed out, raised, or returned nothing usable
    log.warning(score.guidance)
elif score.total < threshold:
    ...                       # a real verdict, and a poor one
    print(score.worst)                 # -> 'accuracy'
    print(score.principles)            # -> {'accuracy': 0.2, 'tone': 0.9}

Three things that came out of integrating it rather than reading it:

• The judge is a network call, so it fails the way networks fail. A judge that raises — a 503, a reset connection — used to propagate straight out, so the commonest failure in production was the one path that did not fail closed. It is now an evaluation failure like any other.
• A hung provider used to hang the caller for as long as its socket stayed alive. There is now a deadline (timeout_s, default 20s), and exceeding it fails closed.
• score.principles and score.worst carry the per-principle numbers the judge already returned. Previously only a single total came out, so a caller knew a draft failed but not which principle failed — nothing to target a regenerate with, nothing specific to tell anyone. A failed evaluation carries an empty breakdown rather than an invented one.

asyncio.CancelledError is re-raised, never swallowed.

The redaction bug worth knowing about

A PII redactor that blanks out phone numbers will, by default, blank out the business's own phone number — turning "call us on 01234 567890" into "call us on [redacted:phone]". The assistant becomes useless at the thing it is asked most often.

layer = SafetyLayer(allow=["01234 567890", "hello@theshop.co.uk"])
layer.screen("Call us on 01234 567890, not 07700 900999").text
# 'Call us on 01234 567890, not [redacted:phone]'

Published contact details survive; a stranger's number does not. This is obvious in hindsight and was not obvious in advance.

Crisis detection: the default is the feature

CrisisProtocol fires on distress and breaks persona. The part that took the work was not firing — "I need to dye my hair" must not trigger a safeguarding response. A false positive here derails an ordinary conversation, which is its own kind of harm.

In the system this came from, the protocol is on by default and must be explicitly opted out of, rather than opted into. That ordering is the actual safety property.

What this is not

Stated plainly, because the alternative is letting you find out yourself:

• Not a complete safety system. These are three narrow controls. Prompt injection, jailbreaks, training-data leakage and tool-use authorisation are all out of scope.
• Not battle-tested at scale. It runs in one production system with a small user base. It has not been attacked by anyone competent.
• Not novel. LLM-as-judge, PII regexes and crisis keyword detection are all prior art. The contributions here are the conditional trigger and the decorrelation requirement, both of which are engineering judgement rather than research.
• The regexes are first cuts. PII detection by regex is inherently incomplete. The high-stakes trigger is English-only and will miss idioms it was not written for.

Changelog

• 0.3.0 — a judge that raises or hangs is now an explicit failed evaluation, not an exception through the caller or an unbounded wait (timeout_s, default 20s). Score carries principles and worst, so a caller can see which principle failed.
• 0.2.0 — a failed evaluation is no longer a passing one: _finite01 clamped an out-of-range score UP to 1.0, and an unparseable reply scored a neutral 0.5. Both now produce Score(total=0.0, evaluated=False).

Provenance

Extracted from PANTHEON, a private multi-tenant agent substrate, in September 2026. The git history is preserved from the original commits — the first is phase 00: bootstrap, 5 June 2026. The extraction removed a dependency on the product's global settings object; the judge is now injected instead of built from a configuration singleton.

The tests were rewritten during extraction. The originals exercised these classes through the agent loop, which meant they tested the loop as much as the guardrail — if a behaviour can only be demonstrated by standing up a whole runtime, it isn't really a library.

Development

uv venv && uv pip install -e ".[dev]"
python -m pytest

Who built this

Isaac Teague Frayling — pantheonlabs.co.uk · github.com/Igfray

Extracted from PANTHEON, a governed multi-tenant substrate for running AI agents against other people's money and data. The other pieces published from it: pantheon-rls (tenant isolation as a Postgres guarantee), pantheon-credit-ledger (overdraft-proof metering), pantheon-ssrf-guard, pantheon-tool-sanitizer and pantheon-ical.