Cerberus

Watches API-key usage metadata and tells you when one of your customers' keys starts behaving unlike its own history — a shared, stolen, or scraped credential — before the bill arrives.

Not bot detection. Specifically: this paying customer's key is doing something it has never done, and it's costing you money.

---

What Cerberus receives

Twelve fields. Nothing else. The ingest endpoint rejects any unknown field by name, so this list is enforced in code rather than promised in prose.

Field	Meaning
ts	ISO 8601 UTC timestamp
key_fp	Fingerprint of your API key ID — never the key
endpoint	Route template (see below)
tokens_in / tokens_out	Token counts
latency_ms	Request latency
status	HTTP status code
ip_fp	Fingerprint of the client IP — never the address
ip_net_fp / ip_block_fp	Fingerprints of the surrounding network
ip_family	v4 or v6
cost	Optional, in USD

No content, ever. No prompts, no responses — not hashed, not truncated, not optional.

Thirty-day rolling retention.

Why we cannot see your keys or your users' IP addresses

Fingerprints are HMAC-SHA256 under a secret salt you generate and we never receive, truncated to 128 bits.

This matters because a plain hash is not anonymisation: the IPv4 space is 2^32, so a GPU builds the whole rainbow table in minutes. With a salt we never hold, we cannot reverse a fingerprint even in principle — and because the salt is per-tenant, the same IP hitting two of our customers produces two unrelated fingerprints. We could not build a cross-customer profile if we wanted to.

Salt handling

• Store it wherever your other secrets live, not in a config file.
• Use the same salt on every instance, or one key looks like several.
• Do not rotate it. Every historical fingerprint becomes uncorrelatable and all baselines reset.
• If you lose it, baselines reset. Nothing else breaks.

Looking up a fingerprint at 2am

An alert names a key by fingerprint, because that is all we have. You can always map it back — you hold both the salt and your key list:

from cerberus_keys import fingerprint
{fingerprint(k, salt): k for k in my_api_keys}

We cannot do this lookup. That is the point. Build the index once during onboarding and keep it.

Installing

pip install cerberus-keys

Zero runtime dependencies, standard library only.

from cerberus_keys import Cerberus

client = Cerberus(ingest_token, salt, endpoint_url="https://.../v1/events")
client.record(api_key=key, ip=client_ip, endpoint="/v1/chat",
          tokens_in=n_in, tokens_out=n_out, latency_ms=ms, status=200)

Is it actually recording?

client.health
# {"ok": True, "dropped": 0, "rejected": 0, "invalid": 0,
#  "queued": 0, "last_rejection": None}

ok is the one thing worth putting on your own status page. It goes False when the client is discarding data for a reason that will not fix itself:

• rejected — the server refused a batch permanently (bad token, malformed field, an event past the backfill window). Retrying cannot help.
• invalid — the client could not even build the event. The usual cause is ip not being a bare address: a port, an unknown, an RFC 7239 for= value or a bracketed IPv6 will all do it. See the gateway section above.

dropped is different and is not an error: it means the buffer was full and the client shed load rather than slowing your request path. queued includes any batch in flight, so dropped + rejected + invalid + queued accounts for everything you recorded.

record() never blocks, never raises, and never adds latency to your request path. It enqueues and returns; a full buffer drops rather than applying backpressure; every exception is swallowed at the boundary. A monitoring library that can stall the thing it monitors is not worth running. Dropped events are visible on client.dropped and are never an error.

If you run a gateway, proxy or load balancer, read this first

ip must be the real client address, not the address of your own infrastructure. This is the one misconfiguration that leaves Cerberus installed, reporting healthy, and unable to ever fire.

Behind any load balancer, ingress or CDN, the socket peer your framework reports (request.client.host, req.socket.remoteAddress, REMOTE_ADDR) is the load balancer, identically on every request. Every key then has exactly one distinct address forever, the fan-out rule compares one against one, and no leak can produce a deviation. Events arrive, /v1/status says the key is watched, and nothing can ever alert.

Take the leftmost entry of X-Forwarded-For, which is the original client:

from cerberus_keys import client_ip_from_forwarded_for

ip = (client_ip_from_forwarded_for(request.headers.get("x-forwarded-for", ""))
      or request.client.host)
client.record(api_key=key, ip=ip, endpoint=route_template, ...)

Send the leftmost entry only. The header is a chain (client, proxy1, proxy2) and fingerprinting the whole string counts every hop combination as a separate address, which makes fan-out counts meaningless in the other direction.

On LiteLLM this is a config flag, not code — see the section below; it is the second of the two required blocks and the one that is easy to miss.

How to check you got it right. GET /v1/status reports what Cerberus can see per key. A key sitting at one distinct address is reported as below the detection floor. Cerberus also checks each account for the whole-fleet version of this shape and tells you unprompted once enough keys share one address — but that notice has a threshold, so on a small account the /v1/status reading is the check that matters. Don't rely on either; set the header.

You will know within the hour that it works

Once your first events reach us, Cerberus posts a one-time confirmation to your Slack webhook. It arrives on the next hourly pass — so within the hour, not instantly — and it tells you:

• how many events we received, and how many distinct API keys they came from
• how many distinct client IPs we can see
• when detection starts, since we need about a week of history per key first

If it does not arrive, something is wrong, and that is the point. Cerberus can fail in a way that looks exactly like working: this SDK drops failed sends silently, on purpose, so it can never add latency or raise into your request path. A Cerberus that is receiving nothing looks identical to a Cerberus that is watching quietly. The confirmation exists so that silence means something specific instead of nothing.

If an hour passes and no message arrives, check in this order:

1. Is a Slack webhook configured on your account? No webhook, no message — and no alerts either.
2. Is the ingest token right, and not revoked? A bad token gets a 401, which the SDK drops.
3. Is endpoint_url correct? Anything non-2xx is dropped the same way.
4. Is your process alive long enough to flush? The client batches. A script that exits immediately should call .close().
5. Is ip a bare address? Behind a proxy it often is not — a port, an unknown, an RFC 7239 for= value. Every such event is discarded before it reaches the network.

client.health["ok"] is the fastest local check, and it is False for exactly the two causes that never fix themselves: rejected (the server refused) and invalid (the client could not build the event). Reading dropped alone will not tell you — a full buffer is healthy shedding, and until 0.3.0 a malformed address was counted there too, so a gateway could discard every event while ok stayed True.

endpoint must be a route template

Send /v1/orgs/{org_id}/chat, not /v1/orgs/acme-corp/chat. Live paths routinely carry identifiers, and this field would carry them to us.

The ingest endpoint rejects paths containing @, UUIDs, or long digit and hex runs — but that is defence in depth, not a guarantee. /v1/orgs/acme-corp/chat defeats every one of those checks. The template requirement is the mechanism; we cannot detect every violation of it.

Importing history so detection starts today

Cerberus will not judge a key until it has enough of that key's own history: 500 requests across 40 active hours from 3 or more addresses, inside a rolling 7 days. Live traffic takes about a week to get there. If you already have request logs, import them and the baseline closes immediately — the import is rolled into hourly aggregates synchronously, so the keys are watched the moment it finishes.

Pass backfill=True and use a separate client for the import:

importer = Cerberus(
    ingest_token, salt,
    endpoint_url="https://api.cerberushq.dev/v1/events",
    backfill=True,
)

for row in historical_rows:          # oldest first
    importer.record(
        api_key=row.api_key, ip=row.client_ip, endpoint=row.route_template,
        tokens_in=row.tokens_in, tokens_out=row.tokens_out,
        latency_ms=row.latency_ms, status=row.status, cost=row.cost,
        ts=row.timestamp,            # timezone-aware, historical
    )

importer.flush(timeout=300)
importer.close(timeout=60)

# ALL FOUR COUNTERS. `rejected` counts only batches the server refused
# permanently, which is the one way a real import does not fail -- a dropped
# connection or a killed process shows up in `dropped` or `queued`, and an
# event the client could not build at all shows up in `invalid`. Asserting on
# `rejected` alone passes on every realistic half-landed import.
h = importer.health
assert (h["rejected"] == 0 and h["dropped"] == 0
        and h["queued"] == 0 and h["invalid"] == 0), (
    f"partial import, do not trust the baseline: {h}")

A half-landed import is worse than none: the keys still become "watched" and nothing reports that the baseline is thin. Re-running the whole import is safe — rolling an hour recomputes it from events rather than adding to it.

Import 8 days, not 30. Only the trailing 7 days form the baseline. Older events are accepted and stored — up to 90 days, matching how long aggregates are kept — but they sit outside the window and change nothing about detection. Importing a month costs you the upload and buys nothing over importing a week.

Ordinary batches are refused past 48 hours. That is what the flag changes, and it is why passing historical timestamps without it fails: every batch comes back 400 naming the fix. Check health["rejected"] when the import finishes — a backfill that half-lands leaves a baseline computed from partial history, which is worse than no baseline at all.

Do not reuse the importer for live traffic. Every batch it sends carries the flag, which suppresses the lateness marking live events rely on to be re-rolled. One instance is an importer or it is not.

An import never pages you. The hours inside it are not evaluated, so nothing in your history can raise an alert — a month-long import would otherwise page you once for every incident in it. The exception is the most recent closed hour, which the next hourly pass evaluates like any other.

It inherits whatever was happening. If a key was already being shared during the window you import, that becomes its normal and Cerberus will not flag it until it gets worse still. This is not a property of importing — waiting seven days does the same thing, because those days also contain the abuse — it is a property of learning a baseline from history at all.

If you run LiteLLM

This is the cheapest integration on offer: LiteLLM's callback payload already carries eight of the twelve fields, so there is no instrumentation to write. It is two config blocks, and both are required.

First, a file the proxy can import. Put it next to your config:

# cerberus_callback.py
import os
from cerberus_keys.litellm import CerberusLogger

cerberus = CerberusLogger(
    ingest_token=os.environ["CERBERUS_INGEST_TOKEN"],
    salt=bytes.fromhex(os.environ["CERBERUS_SALT"]),
    # Note the `api.` subdomain; this is the endpoint_url from your signup
    # response. The apex is the website and does not accept events.
    endpoint_url="https://api.cerberushq.dev/v1/events",
)

Block one registers it:

litellm_settings:
  callbacks: cerberus_callback.cerberus

Block two is the one that matters, and it is in a different section:

general_settings:
  use_x_forwarded_for: true   # REQUIRED

Without it, LiteLLM reports request.client.host — which behind any load balancer, ingress, or CDN is the load balancer's address on every request. Distinct-IP counts collapse to one permanently and fan-out detection silently cannot fire, while looking exactly like a clean bill of health. It is the only way to install Cerberus, have it appear to work, and get nothing from it.

Cerberus checks for this shape and tells you if it sees it. Don't rely on that; set the flag.

Two things that are different on LiteLLM

Resolving a fingerprint takes the hash, not the key. LiteLLM never hands a callback your raw key — it passes user_api_key_hash, which is sha256(key).hexdigest(). So the fingerprint in an alert is HMAC(salt, sha256hex(key)). Calling resolve() with raw keys matches nothing, silently, and reads exactly like "that key isn't ours" at the worst possible moment. Map them first:

from cerberus_keys import resolve
from cerberus_keys.litellm import key_fingerprint_input

resolve(fp, [key_fingerprint_input(k) for k in your_keys], salt)

Streaming latency is time-to-first-token. LiteLLM's response_time is completion_start_time - start_time when stream=True, not the full duration. Latency is a digest signal rather than a fan-out condition, so this does not affect what pages you — but a streaming-heavy proxy will show lower latencies than its users experience, and that is a property of the source, not of Cerberus.

The proxy path only. requester_ip_address is populated in LiteLLM's proxy-side request handling, so the LiteLLM SDK used as a library does not carry it — take the normal Cerberus(...).record(...) path there.

What it detects

One real-time signal: key fan-out. Not "many IPs" — many IPs each doing very little, which is the shape of a shared credential and what separates it from your infrastructure scaling up. Scaling three containers to forty raises volume in proportion; a leaked key inverts that. Five conditions must all hold, sustained across consecutive hours, before anything pages you.

Everything else — cost and volume anomalies — goes in a daily digest. A customer who just launched and 10x'd their usage looks identical to abuse, and that customer is the best thing that happened to you this quarter.

Known limits, stated plainly

• Keys used from fewer than 3 distinct IPs are not protected at all. A key that lives on one or two servers never builds a baseline the rule can compare against, so a leak of that key -- however dispersed, however sustained -- does not fire. This is the single biggest gap, it covers the most common key shape, and GET /v1/status names the affected keys as below_detection_floor rather than pretending they are still warming up. The floor is a calibration threshold; it will move only on replay evidence, not by guess.
• Keys already spread across many IPs are under-protected. The rule is relative to each key's own history, so a key that normally lives on 200 IPs needs a far larger jump to trip. Conservative by design; you should know it.
• A leak that ramps slowly enough is never caught. The baseline tracks it upward and nothing fires. Detecting that needs a long-horizon reference we don't yet have.
• Concentrated datacenter scrapers don't page. A scraper rotating inside one provider's range looks concentrated, and the rule requires dispersion. Deliberate: a missed scraper is cheaper than a false page.
• A serverless migration can page you. If you move a key from a few servers to many small workers (a serverless or autoscaling rollout) at roughly constant traffic, and those workers are spread across many network blocks — which cloud egress usually is — that has the same shape as a leaked key being used from many places, and the rule can fire. It is the one false alarm the design cannot rule out from the data it holds: it cannot tell your cloud's address ranges apart from a stranger's. If it happens, acknowledge the alert (it offers a "this key is legitimately distributed" link) and it stops. Tell us if you have a rollout planned and we can quiet the key ahead of time.