leakmap

See what personal data your app sends to third parties. Fail CI when a new flow appears.

Run your tests through leakmap. It records every outgoing request, finds personal data in it (Aadhaar, PAN, UPI IDs, Indian mobile numbers, emails, cards and more), names the company and country receiving it, and keeps a reviewed baseline in your repo. A pull request that starts sending something new fails the check.

Try it (10 seconds, offline)

uv tool install leakmap
leakmap demo --open

The demo runs a small fake support bot that calls OpenAI, Langfuse, Sentry, Mixpanel, Razorpay and an unknown CRM. Every request is answered locally, so nothing leaves your machine.

Use it on your app

leakmap run -- pytest              # record what your tests send out
leakmap report --format html -o leakmap.html
leakmap baseline                   # accept current flows into leakmap.lock, then commit it
leakmap check                      # exit 1 if a run has flows not in leakmap.lock

In CI, one step does both:

- run: uv tool install leakmap
- run: leakmap run --stub --check -- pytest

--stub answers every request locally, so CI never sends test data to real vendors. Leave it off to forward requests normally.

What it finds

A value is reported on its own only when a checksum, registry or distinctive shape backs it. Values without one (a bank account, a CVV, a passport number) are reported only next to a label that says what they are: passport no., CVV:, "routing_number": ....

Names and addresses in free text need a language model. uv tool install 'leakmap[ner]' and leakmap run --ner add GLiNER's PII model (Apache-2.0, runs on CPU, pinned to a commit). It adds torch to the install and 0.1–0.5 s per long text value, so it is off by default.

Base64 values are decoded and scanned (Mixpanel sends events as data=<base64 JSON>). It looks inside JSON, JSON encoded inside JSON (common in tracing SDKs), NDJSON / Sentry envelopes, form bodies, multipart, query strings, URL paths and OpenTelemetry protobuf (Langfuse, Logfire, Arize, Traceloop). Detectors live in leakmap/data/detectors.yaml; vendors in leakmap/data/vendors.yaml (127 companies).

Precision comes first: a check that raises false alarms gets removed from CI. The test suite includes everyday payloads (UUIDs, trace IDs, timestamps, order IDs, tool schemas) that must produce no findings.

Accuracy

Scored on four public PII datasets I did not write: Nemotron-PII, Gretel finance, Ylemis and Maskara, 133,194 test documents in all. The rules were frozen before any test set was scored, and the numbers below come from that one run, at the default floor. "Valid" means gold values that pass the category's checksum. Generators also write numbers no real system issues, and leakmap drops those on purpose.

None of the 1,956 negative documents in Ylemis and Maskara produced a finding.

Weak spots:

• Names. Without --ner, a name is found only after a label or in a name field. With --ner, name recall is 72–99%, but precision drops to 59–100% depending on the dataset (details).
• Addresses are only partly found.
• Bare IPs. At the default floor, a bare IP is hidden as low confidence.
• Hindi and Bengali labels. Voter IDs labelled in those scripts are missed.

Several of the "false alarms" are real data that the dataset left unlabelled, such as session cookies, diagnoses and ZIP codes. Rule bugs that the held-out run turned up were fixed afterwards, and those numbers are reported separately as tuned on test.

Full per-category tables with 95% intervals, every dataset, the label-quality checks and the error breakdown are in benchmark/independent.md.

Older checks: the synthetic Indian API traffic set (benchmark/) is one I wrote, so treat it as a regression suite. Real traffic recorded in the pydantic-ai, crewAI, langchain and llm-anthropic test suites (5,109 requests) was checked by hand. In live runs on real codebases with planted values, all 27 that left the machine were caught. That is a small sample.

Tested against real SDKs

tests/test_real_sdks.py runs the official SDKs through the proxy with fake values and checks what is found:

Rules (leakmap.yaml, optional)

home_country: IN          # flows outside this country are highlighted
min_confidence: medium    # low | medium | high
ignore_hosts: ["*.internal.example.com"]
deny:                     # always fail, even if baselined
  - data: aadhaar
    country_not: IN
    reason: Aadhaar must not leave India

How it works

leakmap run starts mitmproxy on 127.0.0.1 and runs your command with HTTPS_PROXY and a temporary CA set for Python, Node (NODE_EXTRA_CA_CERTS, NODE_USE_ENV_PROXY), curl and Go. Each request is inspected in memory; only the category, field path and a masked sample (AB******4F) are written. Bodies, headers and credentials are never stored, and the CA is deleted when the run ends.

Limits

• It sees requests that reach the network. If your tests mock HTTP clients, those calls are invisible; run an integration suite or a staging smoke test instead.
• Browser-side trackers (pixels, analytics JS) are not covered yet; this is for backend and test traffic.
• In --stub mode every request gets {} back. SDKs that validate responses may log errors.
• Country is the vendor's headquarters. Where your data is processed depends on your plan and region; check the vendor's DPA.
• A clean report means leakmap saw nothing in this run, not that nothing is sent. It is evidence for a privacy review, not legal advice or a compliance certificate.

License

Apache-2.0