foxy-audit (SDK)

Governance-as-Code for AI. One decorator -> a tamper-evident, content-blind audit trail.

The SDK creates customer-keyed HMAC commitments for supported LLM inputs and outputs locally, throws raw text away before upload, and durably spools only metadata to the Foxy Audit backend. It also fires a best-effort local UDP ping so the desktop "fox" companion shows local capture activity and backend grading alerts.

1.6.0 — every policy tag now runs the baseline checks, and hipaa_basic is real

If you run policy="hipaa" or policy="gdpr" under mode="block" or mode="redact", prompts that used to pass may now be blocked or rewritten. That is the fix, not a regression.

Two defects, one release, because fixing either alone makes the other worse.

The policy map replaced instead of adding. hipaa ran the PHI sweep instead of the prompt-injection and secret-key checks — so the HIPAA workspace was the one workspace that did not notice an API key pasted into a prompt. The map is now additive: injection and secret detection run under every tag, and a domain tag adds its personal-data family on top.

hipaa_basic was not a tag at all. It is the tag in this README's own quickstart, in the package docstring and in demo/run_demo.py — and it was not a key in the policy map, so it fell through to the default and ran zero PHI detection. The event still shipped tagged hipaa_basic, still entered the hash chain, and still appeared in the Compliance Passport, which groups its statistics by policy_tag. A customer following our own quickstart got a document attesting activity under a HIPAA-named policy that had never performed a HIPAA check. hipaa_basic and gdpr_basic are now aliases for hipaa and gdpr.

Aliasing alone would have been the other half of the same bug — hipaa_basic would have gained PHI and lost injection and secrets. Only the additive baseline makes it safe, so both ship together.

What this changes for you, by mode:

• New prompt_injection / secret_key labels appear in pii_signals, and new injection.* / secret.* ids in policy_rules, on hipaa/gdpr rows.
• Your breach count does not rise from those labels. The only rows that gain them are blocked and redacted rows, and the backend grades those from their enforcement labels via policy_engine.evaluate_enforcement, which never reads pii_signals. One classification does move: under hipaa/gdpr, a prompt tripping only an injection or secret rule now produces a terminal host-enforced row (policy_breach false, risk 0) instead of a judge-graded one. A prevented egress is not a breach.
• An unrecognised tag now warns instead of silently degrading in silence. It still runs and it still ships — see Policy tags below.
• policy_tag on the wire is unchanged. It is recorded exactly as you passed it: hipaa_basic still reads hipaa_basic in the ledger and in the Passport. Only the checks resolve through the alias, so the meaning of every historical row that used the tag is untouched.

1.5.0 — mode="redact" now examines the response for PII

If you run mode="redact", your rows will carry more pii_signals labels than they did on 1.4.x, and you should expect that.

Until 1.5.0, a redact-mode call whose prompt tripped the policy reported only the labels that fired on the prompt. The prompt+response PII sweep was skipped entirely on those rows, so PII the model returned in its response was never recorded — in the one mode chosen specifically because the customer cares about PII. observe mode, which promises less, always got the full sweep.

pii_signals is now the union: exactly what fired on the prompt, plus everything the sweep finds across prompt and response, deduplicated and sorted.

What this changes for you:

• More labels on redact rows, including PII kinds the prompt never contained.
• Your breach count does not move. A redacted row is a terminal, host-decided event, and the backend grades it from its enforcement labels — pii_signals is not a breach trigger on that path, on either the chained verdict or the graded one. (Measured, and pinned by backend/tests/integration/test_blocked_events.py::test_a_redacted_rows_pii_signals_do_not_make_it_a_breach.)
• New rows hash differently from old ones. pii_signals is chain material, so a row recorded on 1.5.0 covers labels a 1.4.x row would not have. Existing rows and their chain are untouched, and verification of both is unaffected.

There is no flag to turn this off. On an audit product, an opt-out from correct detection is a setting whose only use is making the evidence say less than the system knows.

Install

pip install -e .            # from this sdk/ folder, for local development

Runtime dependency: requests only.

Use

import os
from foxy_audit import FoxyClient

foxy = FoxyClient(api_key=os.getenv("FOXY_API_KEY"))   # or just rely on the env var

@foxy.audit(policy="hipaa")
def ask_model(prompt: str) -> str:
    return llm_client.generate(prompt)     # your existing code — unchanged

Every call to ask_model is now hashed, logged, and graded. Or use the module-level decorator, which builds a client from the environment:

from foxy_audit import audit

@audit(policy="soc2")
def summarize(text: str) -> str:
    ...

Policy tags

policy selects which local checks run before the model is called. The map is additive — the baseline runs under every tag, and a domain tag adds to it.

An unrecognised tag runs the baseline and warns. policy_tag is a free string on the wire — the backend validates no vocabulary, and labelling rows in your own terms ("claims_triage", "internal_v2") is supported and normal. So a tag we do not know is not an error and will not raise: failing your production model call over a label would be a worse outcome than the label being unknown. But it is no longer silent, because that was the actual defect — a typo like "hipa" would quietly downgrade a workspace's compliance posture with nothing said anywhere:

UserWarning: foxy-audit: unrecognised policy tag 'hipa'. Running the baseline checks
only (prompt-injection + secrets); NO PHI/PII check will run. Known tags: default,
gdpr, gdpr_basic, hipaa, hipaa_basic, soc2.

The warning fires once per distinct tag per process. Whatever you pass is recorded on the wire verbatim, recognised or not.

Attributing the model (agent)

Pass agent= to record which model produced the interaction. The backend folds it into the tamper-evident hash chain, so the attribution can't be altered after the fact:

@foxy.audit(policy="soc2", agent="gpt-4o")
def ask_model(prompt: str) -> str:
    ...

agent is optional — rows logged without it hash exactly as before, so existing chains keep verifying.

Scanning the response (OWASP LLM05 — Improper Output Handling)

mode governs the prompt. response_scan governs what came back: markup that will be rendered, a SQL statement the caller might execute, an SSRF-shaped URL, a secret the model echoed, and — under hipaa/gdpr — personal data the model returned that the prompt never contained.

foxy = FoxyClient(api_key=..., response_scan="block")   # or FOXY_RESPONSE_SCAN=block

It never rewrites a response. There is no response-side "redact" under any mode, and mode="redact" scans the response exactly as observe does. Prompt redaction changes what the model sees; response redaction would change what your parser, database and UI receive — and for a provider response object, which is not a string, it would silently do nothing at all.

On a streamed response, blocking is partial. This is architectural, not a bug. A generator hands each chunk to you as it arrives and a chunk cannot be un-yielded. Under response_scan="block" each chunk is scanned before it is yielded, with a 256-character carry-over window so a match split across a boundary is still caught, and the stream is terminated at the first flagged chunk — so the rest never arrives, but everything already yielded has already been delivered. Two bounds follow: a pattern whose halves land further apart than that window is not prevented, and the scan adds per-chunk latency. A completed stream is re-scanned whole in both modes, so the evidence record is exact even where prevention was not. Buffering the whole stream would make blocking total and would silently turn a streaming API into a non-streaming one, so the SDK does not do it.

A cut stream is recorded as truncated, never as prevented — whatever mode you run. Chunks you already received are in your application, so calling that "prevented egress" would put a false statement in your Compliance Passport. Only a block where nothing reached you is recorded as event_type: response_blocked; a stream cut after delivery is an ordinary stream event with decision: response_truncated, and the exception says so in as many words. One consequence worth knowing: if mode="redact" scrubbed the prompt and the stream is then cut, that row is not counted in the Passport's redaction tally — one row carries one terminal outcome, and this one's is truncation. The redaction is still in the record, as the phi.*/pii.* rule ids that fired.

What the scan reads, and what it admits it cannot. Response content is extracted from the provider's own shape — OpenAI choices[].delta.content / choices[].message.content, Anthropic content blocks and delta.text, the Gemini candidates[].content.parts[], the Responses API output[]/output_text — plus plain strings and bytes (decoded UTF-8, errors="replace", so raw SSE is covered). An unrecognised but serialisable shape is scanned as a serialised envelope and recorded as response_scan.degraded; an object whose content cannot be reached at all is recorded as response_scan.unreadable. Neither ever blocks — coverage you do not have is missing evidence, not a finding — but neither is silently reported as a clean scan. They appear only as rule ids: never as a decision, never as a blocked_reason, and never in the Passport's enforced-rule table. "We could not read this" is not a verdict on the interaction.

audit_required does not hide a block. If the audit event cannot be durably delivered, you still get FoxyResponseBlocked, with audit_delivery_failed=True on it. The security decision outranks the delivery guarantee.

Upgrading from 1.3.x changes nothing you receive. The default detects and records; it never raises and never rewrites. A response that trips nothing emits the identical payload it emitted before. Turning on prevention is a deliberate response_scan="block".

These are regexes, not a parser — a model explaining SQL will trip response_sql.destructive. That is exactly why prevention is opt-in.

Configuration

Salted commitments (optional)

Set salt_sidecar_path and each event gets a fresh 128-bit salt, mixed into the HMAC canonically — HMAC(key, {"s": salt, "v": <canonical text>}), never by concatenation. The salt is appended to that local JSONL file and never leaves your process: not the wire, not our database, not a response, not a log line. Those rows report commitment_alg: "hmac-sha256-salted"; leave the setting unset and commitments are byte-identical to what the SDK has always produced.

The trade: lose that sidecar and you lose the ability to prove which text those commitments cover (foxy_verify.py --commitment-key --events reports them as not checked). You keep chain verification either way — tamper-evidence is recomputed from stored fields and never needs the salt.

With no API key the SDK is a graceful no-op for the cloud path: it still runs your function and still pings the desktop fox, but skips the HTTP upload. In the default mode, delivery is best-effort; for regulated workflows, set audit_required=True so the decorator waits for a server receipt and raises when durable delivery cannot be confirmed.

Guarantees

• Default path is asynchronous — the HTTP upload runs on a background daemon thread after a local durable enqueue.
• Retries do not discard events — failed uploads remain in the SQLite/WAL spool.
• Content-blind by design — commitments, token counts, policy tags, and bounded identifiers leave the host; raw text is not sent by the SDK. The response scan is no exception: it emits rule ids such as response_markup.script_tag, never the matched text, never an offset, never a length.
• Works with sync, async, sync-generator and async-generator functions. Host return values are passed through unchanged — the one exception is response_scan="block", which raises FoxyResponseBlocked instead of returning a flagged response, and which is off unless you turn it on.

What gets sent

To the backend (POST /v1/logs, Authorization: Bearer <key>):

{"event_id": "<uuid>", "client_id": "...", "client_seq": 1, "commitment_alg": "hmac-sha256", "prompt_hash": "<64 hex>", "response_hash": "<64 hex>", "token_count": 123, "policy_tag": "hipaa_basic"}

To the desktop fox (UDP 127.0.0.1:9999):

{"event": "hash_ok", "policy": "hipaa_basic", "tokens": 123, "ts": 1719300000}
{"event": "policy_breach", "reason": "...", "risk_score": 87, "policy": "hipaa_basic", "ts": ...}