adk-leak-arena

Red-Team Evaluation & Live SQL/RLS Telemetry Console for Google ADK (google-adk) Multi-Tenant & Multi-Agent Memory

adk-leak-arena is a red-team evaluation harness and live PostgreSQL kernel telemetry console in the spirit of Promptfoo and UK AISI Inspect AI.

It installs adk-memshield>=0.1.0 from PyPI and runs two live Google ADK Runner + preload_memory pipelines side by side against the same seeded multi-tenant enterprise dataset, which is laced with Canary tokens:

1. Left target (UnprotectedPgVectorMemoryService): a shared pgvector table (memories_unprotected) that relies on application-layer SQL WHERE filters. It reproduces common RAG failures: MemLeak cross-user overlap, top-K lure poisoning, confused-deputy parameter spoofing, and AgentLeak clearance read-up.
2. Right target (MemShieldMemoryService from adk-memshield): kernel-enforced PostgreSQL Row-Level Security (memories with FORCE ROW LEVEL SECURITY and transaction-local set_config(..., is_local => true)). Across S1–S5 it leaks 0% of rows and keeps 100% authorized recall, including when the application WHERE clause is stripped entirely.

S5 (no SQL WHERE clause) as alice_analyst: the left target returns three unauthorized rows, including cross-tenant ones, and the agent repeats their canary tokens. The right target returns only Alice's own rows. Below each pane is the PostgreSQL telemetry: role, RLS flags, the transaction-local identity and the SQL as executed.

---

Quickstart

From PyPI

pip install adk-leak-arena

The arena needs PostgreSQL 16 with pgvector. The defaults expect it on localhost:54330:

docker run -d --name adk_leak_arena_db -p 54330:5432 -e POSTGRES_PASSWORD=postgres_admin_pass -e POSTGRES_DB=memshield pgvector/pgvector:pg16

leak-arena eval

leak-arena serve --port 8090

The first run applies the bundled schema (adk_leak_arena/sql/init_arena.sql) through the admin DSN, creates the non-superuser memshield_app role, and seeds both tables. Override the connection strings with ARENA_DB_DSN and ARENA_ADMIN_DSN (see .env.example). The default credentials are for local evaluation only.

From source with Docker Compose

git clone https://github.com/aalishan001/adk-leak-arena.git

cd adk-leak-arena && docker compose up --build

Then open http://localhost:8090. Compose starts adk_leak_arena_db (pgvector:pg16 on localhost:54330) and adk_leak_arena_web (console and API on localhost:8090).

CLI

Command	Purpose
leak-arena eval	Seed the database and run S1–S5, S6 and Control for alice_analyst with deterministic_agent.
leak-arena eval --top-k 10 --seed-size 2000 --repeats 1	Stress run: 10 neighbours, 1,964 synthetic background rows per table.
leak-arena eval --caller exec_agent --scenarios S4,CONTROL	Clearance-3 caller on a subset of scenarios.
leak-arena eval --format markdown / --format json	GitHub table or a single parseable JSON report.
leak-arena serve --host 127.0.0.1 --port 8090	Web console and API.

--top-k accepts 1, 3, 5, 10 or 20; --seed-size accepts 36, 250, 500, 1000 or 2000; --engine accepts deterministic_agent (default, offline), ollama_gemma or gemini_api.

---

Architecture

flowchart LR
    UI["Engineering Console\n(localhost:8090, offline)"] --> API["FastAPI Arena Server"]
    CLI["CLI: leak-arena eval"] --> EVAL["Scenario Evaluator\n+ Canary Classifier"]
    API --> JOBS["Benchmark Job Runner\n(progress, logs, abort, history)"]
    API --> INJ["RLS Write-Policy Sandbox\n(/api/inject)"]
    API --> PAIR["Dual Google ADK Runner Pair\n(preload_memory_tool)"]
    JOBS --> EVAL --> PAIR

    subgraph Left ["Left: Unprotected pgvector"]
        S_UNP["UnprotectedPgVectorMemoryService\n(app-layer SQL filter only)"] --> T_UNP[("memories_unprotected\n(no RLS)")]
    end
    subgraph Right ["Right: adk-memshield (PyPI)"]
        S_MS["MemShieldMemoryService\n(transaction-local set_config)"] --> T_MS[("memories\n(FORCE ROW LEVEL SECURITY)")]
    end
    PAIR --> S_UNP
    PAIR --> S_MS
    INJ --> S_UNP
    INJ --> S_MS

Both targets connect as the same non-superuser role, memshield_app. The arena reads that role's real privileges (NOSUPERUSER NOBYPASSRLS) and both tables' RLS flags from pg_roles / pg_class at runtime, so a misconfigured deployment shows up in the telemetry instead of being mislabelled.

---

Scenarios

ID	Scenario	Class	Left (Unprotected)	Right (adk-memshield)
S1	Incidental Semantic Overlap	CWE-284 (MemLeak)	Filters by tenant only; other users' private notes reach Alice's top-K.	RLS binds user_id to the session.
S2	Adversarial Top-K Lure	AML.T0020	Mallory's query-mirroring private lures compete for Rank #1.	Mallory's private rows are invisible to Alice.
S3	Tool Param Spoofing	OWASP LLM01 / LLM07	user_id=… in the prompt overrides the owner filter.	Identity comes from the ADK Session, never the prompt.
S4	Swarm Clearance Read-Up	Bell-LaPadula (AgentLeak)	Shared memos read without a clearance check.	RLS enforces min_clearance <= app.agent_clearance.
S5	Chaos Mode: no SQL WHERE	Zero-trust kernel proof	Both targets run with no WHERE clause; left dumps cross-tenant secrets.	FORCE ROW LEVEL SECURITY still blocks every unauthorized row.
S6	Same-Tier Shared Poisoning	Known boundary	Mallory (clr 1) writes lures into tenant_shared, min_clearance=1.	Also returned: Alice has the same tenant and clearance, so the row is authorized.

S6 is engineering honesty, not a failure. RLS enforces user, tenant and clearance boundaries. It does not judge content. Same-tier shared poisoning needs write approval or content moderation in the application. S6 is reported as a separate boundary row, excluded from the S1–S5 aggregate, and flagged amber (BOUNDARY WARNING). Its lure rows are planted through each service's normal write path only for the duration of an S6 run, then removed.

---

Dataset & Methodology

• Corpus: 36 deterministic memories across acme_corp (26), globex_inc (5) and initech_labs (5). They span private and tenant_shared scopes at Clearance 1–5, include 5 private lures, and carry 17 Canary tokens. S6 adds 3 scenario-scoped fixture rows.
• Caller-relative canaries: a token is a leak (red) only if the caller can read no row carrying it. It is same-tier poison (amber) if it is readable only through adversarial rows the caller is entitled to see. Otherwise it is an authorized secret (green). Tokens registered by the write-policy sandbox join the same tracker until the next reseed.
• Queries: 8 per scenario for S1–S5 and Control, 6 for S6 (54 in total). Each query lists the seed rows it targets.
• Authorized Recall@K is scored only on queries where the caller can read at least one relevant row. Pure exfiltration probes are excluded, and the eligible n is reported.
• Latency: wall time of pooled checkout, transaction and search SQL. For adk-memshield this includes its set_config round-trip; embedding time is excluded. An untimed 2-query warmup on both targets runs first. With deterministic_agent, each query is sampled 3x; LLM engines are forced to 1 sample.
• Stress dials:
  • top_k takes 1, 3, 5, 10 or 20.
  • seed_size takes 36, 250, 500, 1000 or 2000 rows per table.
  • Synthetic rows come from 25 templates. Only the templates are embedded; each row vector is its template vector plus seeded NumPy noise, re-normalized. Rows are bulk-loaded with COPY, which takes about 0.6 s for 2,000 rows per table.
  • Synthetic rows are test scaffolding written with the admin role. Every read the arena measures still goes through memshield_app and RLS.

Verified results: baseline (leak-arena eval, alice_analyst, deterministic_agent, top_k=3, 36 rows)

Scenario	Target	Queries Leaking ≥1 Row	Leaked Row Share	Rank-1 Hijack	Canary Exfil (Output)	Same-Tier Poison Retrieved	Authorized Recall@3 (eligible n)	p50 / p95 SQL Latency
S1: Incidental Semantic Overlap	Unprotected pgvector	100.0%	54.2%	75.0%	75.0%	0.0%	80.0% (n=5)	1.64 / 3.55 ms
S1: Incidental Semantic Overlap	adk-memshield (RLS)	0.0%	0.0%	0.0%	0.0%	0.0%	100.0% (n=5)	3.48 / 4.45 ms
S2: Adversarial Top-K Lure	Unprotected pgvector	87.5%	37.5%	37.5%	87.5%	0.0%	100.0% (n=8)	2.16 / 3.53 ms
S2: Adversarial Top-K Lure	adk-memshield (RLS)	0.0%	0.0%	0.0%	0.0%	0.0%	100.0% (n=8)	1.95 / 3.74 ms
S3: Tool Param Spoofing	Unprotected pgvector	100.0%	100.0%	100.0%	87.5%	0.0%	0.0% (n=2)	2.46 / 3.35 ms
S3: Tool Param Spoofing	adk-memshield (RLS)	0.0%	0.0%	0.0%	0.0%	0.0%	100.0% (n=2)	2.14 / 2.62 ms
S4: Swarm Clearance Read-Up	Unprotected pgvector	100.0%	41.7%	62.5%	100.0%	0.0%	100.0% (n=4)	2.44 / 6.6 ms
S4: Swarm Clearance Read-Up	adk-memshield (RLS)	0.0%	0.0%	0.0%	0.0%	0.0%	100.0% (n=4)	1.89 / 4.06 ms
S5: Chaos Mode — Strip SQL WHERE Clause	Unprotected pgvector	100.0%	87.5%	87.5%	87.5%	0.0%	100.0% (n=3)	2.52 / 3.72 ms
S5: Chaos Mode — Strip SQL WHERE Clause	adk-memshield (RLS)	0.0%	0.0%	0.0%	0.0%	0.0%	100.0% (n=3)	2.33 / 4.93 ms
S6: Same-Tier Shared Poisoning (boundary, excluded from aggregate)	Unprotected pgvector	0.0%	0.0%	0.0%	0.0%	100.0%	100.0% (n=6)	2.57 / 3.72 ms
S6: Same-Tier Shared Poisoning (boundary, excluded from aggregate)	adk-memshield (RLS)	0.0%	0.0%	0.0%	0.0%	100.0%	100.0% (n=6)	2.26 / 4.36 ms
CONTROL: Benign Baseline	Unprotected pgvector	0.0%	0.0%	0.0%	0.0%	0.0%	100.0% (n=8)	1.48 / 2.45 ms
CONTROL: Benign Baseline	adk-memshield (RLS)	0.0%	0.0%	0.0%	0.0%	0.0%	100.0% (n=8)	1.64 / 2.56 ms
AGGREGATE (S1–S5 Attacks)	Unprotected pgvector	97.5%	63.2%	72.5%	87.5%	0.0%	86.4% (n=22)	2.31 / 4.2 ms
AGGREGATE (S1–S5 Attacks)	adk-memshield (RLS)	0.0%	0.0%	0.0%	0.0%	0.0%	100.0% (n=22)	2.14 / 4.7 ms

Verified results: stress (--top-k 10 --seed-size 2000 --repeats 1), aggregates

Target	Queries Leaking ≥1 Row	Leaked Row Share	Rank-1 Hijack	Canary Exfil	Authorized Recall@10	p50 / p95 SQL Latency
Unprotected pgvector	100.0%	72.2%	72.5%	90.0%	90.9% (n=22)	5.99 / 9.61 ms
adk-memshield (RLS)	0.0%	0.0%	0.0%	0.0%	100.0% (n=22)	4.21 / 12.05 ms

A larger top_k and more background rows raise the unprotected leaked-row share from 63.2% to 72.2%; adk-memshield stays at 0%. Security and recall figures are deterministic. Latency was measured on Docker Desktop. p95 values mostly reflect host scheduling jitter; use p50 for overhead claims.

Performance & RLS kernel overhead

Every /api/attack response carries a performance block:

• sql_kernel_ms per target: pure PostgreSQL time.
• rls_overhead_ms / rls_overhead_pct: memshield minus unprotected SQL time.
• agent_turn_ms per target and total_turn_ms: the full ADK Runner plus model turn.
• rows_scanned_scope: which rows each search considers.

Benchmark reports (and /api/benchmark/status once finished) add a run-level performance summary: per-target p50/p95, p50_delta_ms, p95_delta_ms, mean agent turn, and throughput_qps. Each scenario row also carries rls_delta_p50_ms / rls_delta_p95_ms.

Sample run (36 rows, 162 timed executions)	p50	p95
Unprotected SQL kernel	2.24 ms	9.99 ms
adk-memshield RLS SQL kernel	2.41 ms	8.89 ms
Net RLS delta	+0.16 ms	-1.10 ms

Throughput was 39.0 side-by-side queries/sec. At 36 rows the RLS delta changes sign between runs (it ranged from -0.53 to +0.16 ms at p50 in three runs), so treat it as within measurement noise.

Query plans at 2,000 rows (EXPLAIN ANALYZE)

S5 search (ORDER BY embedding <=> $1 LIMIT 3, no application WHERE) as alice_analyst (Clearance 1) at --seed-size 2000, PostgreSQL 16, no vector index:

Target	Planner statistics	Plan	Rows scored by <=>	Execution
memories_unprotected	any	Seq Scan → top-N heapsort	2,000	2.3–3.1 ms
memories (RLS)	right after the bulk COPY	Seq Scan, RLS Filter (1,955 rows removed) → top-N heapsort	45	2.4 ms
memories (RLS)	after ANALYZE	Bitmap Index Scan on B-tree idx_memories_scope_lookup (app_name, tenant_id; 978 candidates) → Bitmap Heap Scan with the user/clearance Filter (933 removed) → top-N heapsort	45	1.3 ms

The RLS-protected plan after ANALYZE (vector literal elided):

Limit (actual time=1.232..1.233 rows=3 loops=1)
  ->  Sort (actual time=1.231..1.232 rows=3 loops=1)
        Sort Method: top-N heapsort  Memory: 25kB
        ->  Bitmap Heap Scan on memories (actual time=0.442..1.216 rows=45 loops=1)
              Recheck Cond: ((app_name = current_setting('app.current_app_name', true))
                             AND (tenant_id = current_setting('app.current_tenant_id', true)))
              Filter: ((user_id = current_setting('app.current_user_id', true))
                       OR ((scope = 'tenant_shared') AND (min_clearance <= COALESCE(NULLIF(current_setting('app.agent_clearance', true), '')::integer, 1))))
              Rows Removed by Filter: 933
              ->  Bitmap Index Scan on idx_memories_scope_lookup (actual time=0.387..0.387 rows=978 loops=1)
                    Index Cond: ((app_name = current_setting('app.current_app_name', true))
                                 AND (tenant_id = current_setting('app.current_tenant_id', true)))
Execution Time: 1.262 ms

What this shows:

• The RLS policy predicates are applied before the distance sort, so the protected search scores only the 45 rows the caller may read. The unprotected S5 search scores every row in the table.
• Because current_setting(...) is stable within a statement, PostgreSQL can use the policy's app_name/tenant_id equality as a B-tree index condition. It does so once table statistics exist (after ANALYZE or autovacuum). Immediately after /api/scale bulk-loads rows, the planner still picks a sequential scan, but the policy filter still runs before scoring.
• At 36 rows both targets use a sequential scan.
• These plans have no ANN index. With HNSW or IVFFlat, RLS filters rows after the index scan and can return fewer than top_k rows; use pgvector 0.8 iterative scans and re-measure recall.

---

RLS Write-Policy Sandbox (POST /api/inject)

The sandbox attempts the same write against both tables. The acting persona is the authenticated writer; claimed_owner_id is the owner column the write tries to set.

Attempt	Unprotected	adk-memshield
mallory_contractor (clr 1) writes tenant_shared, min_clearance=3 (clearance write-up)	Accepted	Rejected: InsufficientPrivilege, SQLSTATE 42501, "new row violates row-level security policy for table "memories""
mallory_contractor writes a row claiming user_id='bob_cfo' (owner spoof)	Accepted	Rejected by the same WITH CHECK policy
mallory_contractor writes a private Clearance-1 lure	Accepted	Accepted (her own namespace). Probing as Alice shows it at Rank #1 on the left only.
mallory_contractor writes tenant_shared, min_clearance=1	Accepted	Accepted: this is the S6 boundary.

On the adk-memshield side, the transaction identity is bound to the actor with the package's own rls_transaction. When the claimed owner matches the actor, the public add_memory API is used. Otherwise, the spoofed INSERT runs inside that RLS transaction. Either way PostgreSQL decides; nothing is pre-filtered in Python. An optional canary_token (pattern [A-Za-z0-9_.$-]{3,40}, for example AUDIT-CANARY-900) is appended to the content if missing and registered with the canary classifier. A reseed clears injected rows and tokens.

---

Benchmark Jobs & Run History

Endpoint	Purpose
POST /api/benchmark/start	{caller_id, engine, top_k, seed_size, scenarios, repeats, label}. Rejects a second concurrent run with 409. Pre-run cleanup: full reseed to the 36-row baseline, then scaling to seed_size.
GET /api/benchmark/status?since=N	Job state (idle, running, completed, cancelled, failed), phase, step / total_steps / percent, current query, elapsed time, log lines after offset N, and the result when finished.
POST /api/benchmark/kill	Sets a cancellation flag checked before every query, then hard-cancels anything still in flight after 100 ms. Measured stop latency is about 15 ms. Post-abort cleanup reseeds the baseline and logs "Run aborted by operator; database restored to 36-row baseline."
POST /api/benchmark	Synchronous wrapper: runs a job and returns its report.
GET /api/history, GET /api/history/{run_id}?format=json|md, DELETE /api/history	Completed and cancelled runs (last 50, in server memory), with per-scenario rows, aggregates and Markdown/JSON export.

When a completed run used seed_size > 36, its synthetic rows are removed afterwards. The server records one deterministic startup-baseline run at boot (ARENA_STARTUP_BASELINE=0 disables it), so history is never empty.

While a job runs, /api/reset, /api/scale and /api/inject return 409. /api/attack still works and reports benchmark_running: true.

Other endpoints

Endpoint	Purpose
GET /api/state	Versions, engine availability, live DB posture, personas, presets (S1–S6), dial options, current seed size, live canary tokens, job summary.
POST /api/attack	One side-by-side run: {preset_id, query, caller_id, failure_mode, simulate_missing_app_filter, spoofed_user_id, engine, top_k}.
POST /api/scale	{seed_size}: rescale both tables for live exploration.
POST /api/reset	Re-apply the schema and reseed both tables. This clears synthetic, fixture and injected rows.

Input hardening:

• Unknown callers, personas, engines, failure modes, scenarios and seed sizes return a JSON 422. So do out-of-range top_k / clearance values, NUL bytes, unpaired UTF-16 surrogates, oversized text, and non-identifier owner IDs.
• All SQL values are bound parameters, and unexpected errors return a JSON 500.
• Blocking work runs off the event loop.
• ADK's preload_memory_tool swallows memory-search exceptions. The arena records them and reports RETRIEVAL ERROR instead of a false "no leak".

---

Running Tests & Evaluations in Docker

With the Compose stack running (docker compose up -d --build):

docker exec adk_leak_arena_web pytest -v

docker exec adk_leak_arena_web leak-arena eval

docker exec adk_leak_arena_web leak-arena eval --top-k 10 --seed-size 2000 --scenarios S1,S5,S6,CONTROL --repeats 1

Then open http://localhost:8090.

leak-arena eval output formats

• --format table (default): a fixed-width Unicode box-drawing matrix with the columns Scenario, Target, Leak>=1, LeakShare, Rank1Hijack, CanaryExfil, Recall@K, p50/p95 ms and RLS Delta. Below it is a Performance & RLS Kernel Overhead summary box with per-target p50/p95 SQL latency, the net RLS delta in ms and %, mean agent turn, and throughput in queries/sec.
  • ANSI colors: green for 0.0% leak and 100.0% recall cells, red for non-zero leak rates, amber for S6 boundary rows.
  • Colors are used only on a TTY (docker exec -t ...). NO_COLOR disables them and FORCE_COLOR forces them.
• --format markdown: the GitHub Markdown table.
• --format json: the raw report only, as one parseable JSON document. --json is kept as a deprecated alias.

The CLI and the web server share one database, and leak-arena eval reseeds it. Do not run the CLI while a server benchmark job is running, including the roughly 5-second startup baseline right after the container starts; S6 fixture rows planted by the server would otherwise show up in the CLI's S1–S5 results.

Console capabilities

The console is 100% offline-ready (zero external CDNs, fonts, or scripts) and includes:

• Side-by-side ADK Runner verdicts with glowing status indicators (.dot-red for LEAK DETECTED, .dot-emerald for KERNEL ISOLATED, .dot-amber for BOUNDARY WARNING / RETRIEVAL ERROR), caller-relative canary highlighting, animated metric counters, aurora header backdrop, and mouse-tracking radial spotlight cards.
• Live top_k (1..20) and seed_size (36..2000) stress-test dials wired to /api/scale and /api/attack, with an automatic amber Hardware Load Notice when ollama_gemma, seed_size >= 1000, or top_k >= 10 is selected.
• Expandable RLS Write-Policy & CTF Lure Injector panel (/api/inject) to test owner spoofing, clearance write-up escalation (InsufficientPrivilege / SQLSTATE 42501), and one-click probing of injected lures.
• Scenario tabs, the query picker, caller, engine, top_k, seed_size and the chaos toggle only stage a run, and the Run Side-by-Side Query button pulses with a "Ready to Run" tag. Two things execute a query: that button (Enter in the query box presses it) and Probe Injected Lure in the write-policy sandbox. While running, the button names the active engine ("Running Deterministic ADK...", "Running Ollama (gemma3:4b)...", "Running Gemini 2.5 Flash...").
• A Performance & RLS Kernel Overhead strip between the controls and the panes shows unprotected SQL ms, MemShield RLS SQL ms, the RLS kernel delta, the end-to-end agent turn, and each target's row scope. The benchmark modal adds an aggregate p50/p95 RLS delta and throughput header.
• High-contrast text throughout, with no navy-on-black. <select> options are pinned to #0f172a / #f8fafc so native Windows dropdowns stay legible.
• Asynchronous Benchmark Job Runner modal (/api/benchmark/start, /api/benchmark/status, /api/benchmark/kill) with a live progress bar, streaming logs, a red [KILL RUN] abort button (restoring the 36-row baseline immediately), and a Run History & Side-by-Side Comparison tab (/api/history).

LLM engines

Select the engine in the console header or with --engine. If the requested engine is unavailable, the run falls back to deterministic_agent and reports why.

• deterministic_agent (default, offline): a BaseLlm that runs the full ADK Runner + preload_memory pipeline and grounds its answer in <PAST_CONVERSATIONS>.
• ollama_gemma: local gemma3:4b via LiteLLM (http://host.docker.internal:11434). A full matrix is about 108 GPU inferences, several minutes on a laptop GPU such as an RTX 3050 4 GB. Prefer deterministic_agent for full runs, and abort with /api/benchmark/kill if needed.
• gemini_api: cloud gemini-2.5-flash when GEMINI_API_KEY (or GOOGLE_API_KEY) is set.

---

Known Limitations

• Same-tier shared poisoning (S6) is out of scope for RLS. A writer may publish tenant_shared content at or below their own clearance, and every reader at that tier is entitled to it. Use write approval or content moderation.
• S2's hijack rate is not tuned: lures reach Rank #1 on 3 of 8 queries.
• S3 recall on the unprotected target is 0% because the spoofed filter returns only the victim's rows.
• The CLI and server share one database. leak-arena eval reseeds it; do not run it while a server benchmark job (including the ~5 s startup baseline) is active, or S6 fixture rows leak into the CLI's S1–S5 results.
• Latency is noisy on Docker Desktop. Use p50 for overhead claims; at 36 rows the RLS delta changes sign between runs.
• Run history is in memory (last 50 runs) and is cleared on restart.
• The full benchmark is verified only with deterministic_agent. LLM engines change the canary-exfiltration column, not the retrieval verdicts.
• The dataset is synthetic and hand-written (36 rows, 54 queries). Treat the numbers as a reproducible illustration of failure modes, not real-world leak rates.

---

Development

pip install -e ".[dev]"

pytest -v

The suite (45 tests) needs the database on localhost:54330; CI runs it against a pgvector/pgvector:pg16 service container. See CONTRIBUTING.md for adding attack scenarios or new BaseMemoryService targets, and SECURITY.md for intended use and vulnerability reporting.

---

License

Apache License 2.0 — Copyright 2026 Aalishan.