sanctions-screening-benchmark

A reproducible, open benchmark for measuring how well a name-screening engine recovers sanctioned parties from varied name strings, and how well it discriminates against structurally similar strings that are not sanctioned.

Screening engines are usually assessed by asking whether they are deployed and configured. That question is nearly uninformative. This project asks a different one: against which specific classes of name variation does this engine fail?

  PERTURBATION          FAMILY             N   HITS   RECALL
  ----------------------------------------------------------
  invisible_char        adversarial       10      4   40.0%  <-- blind spot
  homoglyph             adversarial       20     10   50.0%
  double_collapse       adversarial       17     17  100.0%
  token_drop            benign            12      0    0.0%  <-- blind spot
  translit              benign            16     16  100.0%
  name_order            benign            20     20  100.0%

The headline recall figure is not the point. The class-wise profile is: it says which failure you have, which tells you who owns the fix. A miss on an adversarial class is an evasion exposure. A miss on a benign class is a data quality problem producing false negatives against ordinary customers.

Why this exists

Public sanctions enforcement records repeatedly describe institutions that had a screening program, had documented policies, and still failed, because a control that exists is not a control that works. There is no open, shared way to measure whether a given screening configuration actually catches the variation it will encounter. Vendors publish accuracy claims against undisclosed test sets. Institutions test against their own historical alerts, which are by construction the cases their current configuration already catches.

This benchmark is an attempt at a common, inspectable, reproducible alternative.

Design commitments

No list data is redistributed. The benchmark fetches the current sanctions list from the publishing authority at generation time. A generated benchmark records the retrieval timestamp and endpoint in its manifest, so a result is always attributable to a specific published list state. See docs/LICENSING.md.

Fully deterministic. Every perturbation is a pure function of (input, seed). Two people running the same command against the same list state get byte-identical benchmarks.

Zero dependencies. Pure Python 3.9+ standard library, including the Jaro-Winkler implementation. It runs on a clean interpreter with no install step, which matters because screening people work on locked-down machines.

Negative controls are mandatory. A benchmark that measures only recall is trivially gamed by an engine that matches everything. Precision is measured against generated near-miss strings, with an explicit statement of what that number does and does not mean.

Honest about its limits. See Limitations. The precision figure is not a production alert-volume estimate and is never presented as one.

Adopters

See ADOPTERS.md. Using this benchmark? Open a PR to add your organisation.

Install and run

From source (no install):

git clone https://github.com/Divine16/sanctions-screening-benchmark
cd sanctions-screening-benchmark

# See every perturbation class and what it models
PYTHONPATH=src python3 -m ssb.cli classes

# Run end-to-end against the bundled synthetic fixture, no network needed
PYTHONPATH=src python3 -m ssb.cli demo

# Build a benchmark from the live OFAC SDN list
PYTHONPATH=src python3 -m ssb.cli generate --limit 500 --out benchmark.json

# Score the reference matcher against it
PYTHONPATH=src python3 -m ssb.cli evaluate benchmark.json --threshold 0.85

From PyPI (once published):

pip install sanctions-screening-benchmark
ssb classes
ssb demo
ssb generate --limit 500 --out benchmark.json
ssb evaluate benchmark.json --threshold 0.85

Or install this repo editable:

pip install -e .
ssb demo

Run the tests:

python3 -m unittest discover -s tests -t .

Scoring your own engine

Any callable of the shape (query: str, candidate: str) -> float in [0, 1] can be scored. See examples/evaluate_your_engine.py.

from ssb.benchmark import Benchmark, score, format_scorecard

def my_engine(query: str, candidate: str) -> float:
    return your_screening_api.similarity(query, candidate)

bench = Benchmark.from_json("benchmark.json")
print(format_scorecard(score(bench, my_engine, threshold=0.85, matcher_name="acme-v4")))

Nothing leaves your machine. The benchmark is a local JSON file and the harness makes no network calls during scoring.

The perturbation classes

Eighteen classes across three families. Full rationale and sourcing in docs/METHODOLOGY.md.

Benign, innocent variation

Adversarial, deliberate evasion

Degraded, upstream system loss

Limitations

Read these before citing any number this tool produces.

1. Negative controls are synthetic. They measure discrimination against structurally adjacent strings. They do not estimate a production false-positive rate, which depends on the customer-name distribution an institution actually sees.

2. Names only. Real screening decisions use date of birth, nationality, document numbers, and address. This benchmark models the name-matching layer in isolation. An engine that scores poorly here may perform acceptably in production with strong secondary identifiers, and the reverse.

3. The common-name problem is out of scope. Where a listed name is also borne by millions of unlisted people, the discriminating work is done by secondary identifiers, not by string matching. This benchmark does not measure it.

4. Perturbation realism is argued, not measured. The classes are grounded in documented naming conventions, published OCR confusion sets, and known Unicode evasion techniques. What is not established is the relative frequency of each class in live data. Do not read the class weights as a risk ranking. See docs/METHODOLOGY.md.

5. The reference matcher is a baseline, not a recommendation. It exists so the benchmark produces a number without a commercial licence.

Contributing

See CONTRIBUTING.md. Additional perturbation classes are the most useful contribution, especially ones grounded in a citable naming convention or a documented evasion technique. Coverage of non-Latin scripts is currently the weakest area.

Licence

Code: MIT. See LICENSE. Sanctions list data is not redistributed; see docs/LICENSING.md.

Citation

See CITATION.cff.