Rate Limiter Plugin

Author: ContextForge Contributors

Enforces global per-user, tenant, and tool/prompt-name rate limits across tool_pre_invoke and prompt_pre_fetch hooks, with an additional per-user/per-server quota for tool invocations. Supports pluggable counting algorithms (fixed window, sliding window, token bucket), an in-process memory backend (single-instance), and a Redis backend (shared across all gateway instances).

Runtime Requirements

This plugin depends on cpex>=0.1.0,<0.2 and imports hook models from cpex.framework. The compiled Rust extension is mandatory; there is no Python fallback implementation.

Hooks

If any configured dimension is exceeded, the plugin returns a violation with HTTP 429. All requests include X-RateLimit-* headers. The most restrictive active dimension is surfaced (e.g. if both user and tenant limits are active, the one closest to exhaustion is reported).

Configuration

- name: RateLimiterPlugin
  kind: cpex_rate_limiter.rate_limiter.RateLimiterPlugin
  hooks:
    - prompt_pre_fetch
    - tool_pre_invoke
  mode: enforce          # enforce | permissive | disabled
  config:
    by_user: "300/m"     # per-user limit across all servers
    by_user_per_server: "60/m" # per-user tool limit for each upstream MCP catalog server
    by_tenant: "300/m"   # shared limit across all users in a tenant
    by_tool:             # per-tool overrides (applied on top of by_user)
      search: "10/m"
      summarise: "5/m"

    # Algorithm — choose one (default: fixed_window)
    algorithm: "fixed_window"    # fixed_window | sliding_window | token_bucket

    # Backend — choose one
    backend: "memory"    # default: single-process, resets on restart
    # backend: "redis"   # shared across all gateway instances

    # Redis options (required when backend: redis)
    redis_url: "redis://redis:6379/0"
    redis_key_prefix: "rl"

    # Backend failure policy (default: "open" — fail-open)
    # "closed" — return HTTP 503 BACKEND_UNAVAILABLE violation when the
    # backend can't be reached (correctness over availability)
    fail_mode: "open"

Configuration reference

Rate string format: "<count>/<unit>" where unit is s/sec/second, m/min/minute, or h/hr/hour. Malformed strings raise ValueError at startup. Counts above 1_000_000 are rejected as a sanity ceiling — anything higher is almost certainly a misconfig or a denial-of-service vector against the memory backend.

Unknown config keys (e.g. a typo like redis_ur) are logged at WARN at engine init alongside the accepted-key list, instead of being silently ignored.

Invalid fail_mode values (e.g. "clsoed") are logged at WARN and fall back to "open" so an operator's typo surfaces instead of silently disabling the hardening they asked for.

Omitting a dimension (e.g. no by_tenant) means that dimension is unlimited — no counter is tracked for it.

by_user and by_user_per_server are independent and may be enabled together. For example, by_user: "300/m" plus by_user_per_server: "60/m" gives each user at most 60 tool calls per catalog server and 300 combined tool calls and prompt fetches within the same tenant context. This prevents one user from multiplying their effective quota without relying on the shared by_tenant ceiling. When by_user_per_server is configured, tool calls without a safe non-empty server_id are blocked with RATE_LIMIT_CONTEXT_MISSING (HTTP 503) rather than bypassing the limit.

Prompt fetches neither require a catalog server_id nor consume by_user_per_server counters, even when a virtual-server ID is present. They still enforce by_user, by_tenant, and prompt-name overrides in by_tool. These counters remain shared with tool invocations within the same tenant context.

Response headers

Every request (allowed or blocked) includes:

Algorithms

Three counting algorithms are available, selected via the algorithm config field.

Fixed window (default)

Counts requests in a fixed time slot (e.g. "minute 14:03"). Resets at the slot boundary. Simple and fast. The 2× burst at a boundary (N requests at the end of slot T, N requests at the start of T+1) is a known trade-off; use by_user with headroom if this matters.

Sliding window

Stores a timestamp for every request in the current window. At each check, expired timestamps are discarded and the remaining count is compared against the limit. Prevents boundary bursts entirely. Memory usage grows with request volume — roughly one float per request per active key.

Token bucket

Each identity (user, user/server, tenant, tool) has a bucket that holds up to count tokens. Tokens refill at a steady rate of count/window. A request consumes one token. Bursts up to the bucket capacity are allowed; sustained rate above count/window is rejected. Useful for APIs where short spikes are acceptable but sustained overload is not.

Redis support: token_bucket with backend: redis is fully supported. The plugin stores {tokens, last_refill} in a Redis hash per key and uses an atomic Lua script to refill and consume tokens in a single round-trip — the same pattern as the other two algorithms. This means token_bucket enforces a true cluster-wide limit in multi-instance deployments.

Backends

Memory backend (default, single-instance only)

• Counters are stored in a process-local MemoryStore (Rust, per-key RwLock — no single global lock)
• An amortized sweep evicts expired keys every ~128 calls — for fixed_window, keys are evicted once the window elapses; for sliding_window, keys with empty timestamp deques are evicted; for token_bucket, keys inactive for >1 hour are evicted
• Limitation: state is not shared across processes or hosts. In a multi-instance deployment (e.g. 3 gateway instances behind nginx), each instance tracks its own counter — the effective limit is N × configured_limit

Redis backend

• fixed_window: atomic Lua INCR+EXPIRE — one Redis round-trip per check, no race condition
• sliding_window: atomic Lua ZADD+ZREMRANGEBYSCORE+ZCARD+EXPIRE — one round-trip, no race condition
• token_bucket: atomic Lua script — reads {tokens, last_refill} hash, refills proportionally, consumes 1 token, writes back — one round-trip, no race condition
• All gateway instances share the same counter — the configured limit is the true cluster-wide limit
• Requires redis_url to be set
• Backend failure policy is governed by fail_mode:
  • "open" (default) — the request is allowed through without rate limiting. Availability over correctness; an infrastructure failure must never block legitimate traffic. Operators should monitor for rate-limiter error logs and treat them as high-priority alerts.
  • "closed" — the request is blocked with a PluginViolation (code BACKEND_UNAVAILABLE, HTTP 503, Retry-After: 1). Correctness over availability; pick this when a failed rate-limit check is less acceptable than a brief outage.

Multi-instance deployment (important): The memory backend is local to a single gateway instance — rate limit counters are not shared across replicas. For multi-instance deployments (e.g., behind nginx or on OpenShift with multiple gateway pods), always use backend: redis to ensure rate limits are enforced correctly across all instances.

Redis TLS configuration

Use rediss:// (double-s) in redis_url to enable TLS. Three levels of TLS hardening are supported:

OS trust store (default for rediss://) — no extra config; Redis's CA must be signed by a CA in the system certificate store:

config:
  backend: redis
  redis_url: "rediss://redis:6380/0"
  by_user: "60/m"

Custom CA bundle — use when your Redis server uses a private CA not in the OS trust store:

config:
  backend: redis
  redis_url: "rediss://redis:6380/0"
  redis_ssl_ca_certs: "/etc/certs/my-ca.pem"
  by_user: "60/m"

Mutual TLS (mTLS) — present a client certificate so Redis can authenticate the plugin:

config:
  backend: redis
  redis_url: "rediss://redis:6380/0"
  redis_ssl_ca_certs: "/etc/certs/ca.pem"
  redis_ssl_certfile: "/etc/certs/client.pem"
  redis_ssl_keyfile: "/etc/certs/client-key.pem"
  by_user: "60/m"

Security note — redis_ssl_check_hostname: false: Due to the underlying redis client API surface, setting this to false disables all TLS certificate validation (both CA chain and hostname), not only hostname verification. A WARN log is emitted at startup. This option is intended only for isolated environments such as local development or integration test rigs. In production, ensure your certificate's CN or SAN matches the hostname instead.

All TLS file paths are validated at plugin init time: missing files and malformed PEM content are surfaced as startup errors rather than at the first request.

Note: The REDIS_SSL_* environment variables used by some Redis clients have no effect on this plugin; use the config keys above.

Tenant-scoped Redis key layout

When the plugin context carries a tenant_id, every dimension key is prefixed with it so counters are isolated per tenant:

rl:{tenant_id}:user:{email}:{window_seconds}
rl:{tenant_id}:user:{email}:server:{server_id}:{window_seconds}
rl:{tenant_id}:tenant:{tenant_id}:{window_seconds}
rl:{tenant_id}:tool:{tool_name}:{window_seconds}

When tenant_id is absent (single-tenant deployments), the tenant prefix is omitted. Global user keys retain the existing layout (rl:user:{email}:{window}); per-server keys use rl:user:{email}:server:{server_id}:{window}.

Upgrade note: the first deploy of the tenant-scoping change causes counters under rl:user:* / rl:tool:* to be orphaned while new writes land at rl:{tenant}:user:*. Counters effectively reset once for all in-flight windows — non-event for typical second/minute windows.

Examples

Single-instance (default config)

config:
  by_user: "300/m"
  by_user_per_server: "60/m"
  by_tenant: "600/m"

Multi-instance with Redis

config:
  backend: "redis"
  redis_url: "redis://redis:6379/0"
  by_user: "30/m"
  by_tenant: "3000/m"
  by_tool:
    search: "10/m"

Sliding window (no boundary bursts)

config:
  algorithm: "sliding_window"
  by_user: "30/m"
  by_tenant: "300/m"

Token bucket — memory backend (default)

config:
  algorithm: "token_bucket"
  by_user: "30/m"   # bucket holds 30 tokens, refills at 30/min

Token bucket — Redis backend (multi-instance)

config:
  algorithm: "token_bucket"
  backend: "redis"
  redis_url: "redis://redis:6379/0"
  by_user: "30/m"

Permissive mode (observe without blocking)

mode: permissive
config:
  by_user: "60/m"

In permissive mode the plugin records violations and emits X-RateLimit-* headers but does not block requests. Useful for baselining traffic before switching to enforce.

Returned Metadata

prompt_pre_fetch and tool_pre_invoke accept an optional extensions parameter carrying OpenTelemetry trace context. When a trace context is present (via extensions.request.trace_id), the plugin emits operational metrics on result.metadata["rate_limiter"] with the following schema:

result.metadata["rate_limiter"] = {
    "allowed": 1,          # int (0/1) — this call's outcome
    "throttled": 0,        # int (0/1) — mutually exclusive with allowed
    "backend": "memory",   # str — "memory" or "redis"
}

allowed/throttled describe only the current call's outcome — the engine evaluates one request per call with no running counter, so the gateway is expected to aggregate across spans/time. All three result branches (no rate limit configured, allowed, throttled) emit metrics identically when a trace_id is present — throttled calls previously emitted no metadata at all; they now carry the same schema as the other branches.

Note: the engine's own operational fields (limited, remaining, reset_in, dimensions) are deliberately not folded into this metrics dict. The gateway's S4 sanitizer only allowlists scalar or list[str] metadata fields — dimensions is a nested dict and can never pass that sanitizer regardless of allowlisting, and remaining/reset_in are not on the sanitizer's numeric allowlist. Emitting fields the consumer structurally can't accept would be misleading, so only allowed/throttled/backend are emitted.

Gating: Metrics are only emitted when a valid trace_id is present in the trace context (extensions.request.trace_id). No trace context means no result.metadata write at all, regardless of any config flag.

Security Note (S1): user_id and tenant_id are never included in result.metadata — they stay confined to PluginViolation.details on the throttled path, a separate channel unaffected by this metrics addition.

Migration Note

Version 0.1.12 fixes prompt retrieval when by_user_per_server is enabled. The per-server quota now applies only to tool invocations; prompt fetches retain existing global and prompt-name limits. Counter keys and tool identity validation are unchanged. Upgrade every replica to 0.1.12 or later before enabling per-server tool quotas with prompt hooks registered.

Version 0.1.11 adds the optional by_user_per_server dimension. Existing configurations and global by_user Redis keys remain unchanged. Enabling the new field creates fresh user/server counters; deploy 0.1.11 to every replica before enabling it so all replicas enforce the same dimensions. Use the stable catalog UUID carried in GlobalContext.server_id; display names or slugs can reset quotas when renamed.

Version 0.1.7 is a breaking change for any existing consumer reading rate-limit metadata:

• The old flat, unconditional result.metadata write (the engine's meta dict — limited, remaining, reset_in, dimensions — written on every allowed/not-limited call regardless of trace context) is now gated on a valid trace_id and replaced by the namespaced rate_limiter key containing only allowed/throttled/backend (the engine's own fields are not folded in — see "Returned Metadata" above for why).
• The throttled branch previously emitted no metadata at all; it now emits the same schema as the other branches when a trace_id is present.
• prompt_pre_fetch and tool_pre_invoke now accept a new optional extensions parameter carrying OpenTelemetry trace context. Emission to result.metadata["rate_limiter"] is gated solely on extensions.request.trace_id being present and valid — if no trace context is supplied, no metrics are written at all, regardless of any config flag.
• Consumers that previously read the flat result.metadata dict unconditionally must migrate to reading result.metadata["rate_limiter"] and must pass a trace_id via extensions to receive metrics.

Lifecycle

The plugin participates in the plugin manager's lifecycle contract:

• async def initialize(self) — invoked once when the plugin manager constructs the plugin. Logs one INFO record naming the active backend (memory / redis).
• async def shutdown(self) — invoked when the plugin manager tears the plugin down (runtime disable, re-instantiation after a config change). Releases backend-held resources — specifically, drops the Rust core's cached Redis multiplexed connection and the SCRIPT LOAD SHA cache. In-flight requests already hold their own clones of the connection and remain valid; the cached reference is replaced on the next request.

Without shutdown, the cached Redis connection would leak across plugin re-instantiation, producing connection churn on the server.

Limitations