custmatch-server: operator guide

custmatch-server runs custmatch as a service next to your data: it keeps your customer records and their loads in one database, matches them in batch on a schedule, answers "who is this?" in real time, and gives data stewards a dashboard to review and correct matches. One image, one database (Postgres in production, SQLite for a trial). How it works is below; what it was measured at is in the report's chapter on the service (docs/report/12-the-service.md).

All processes share the database (CUSTMATCH_DB_URL) and a data directory for run files (CUSTMATCH_DATA_DIR). Run one API process and one scheduler; the API scales to more copies for lookups, but only one holds the write lease for real-time matching at a time.

How it works

                 ┌─────────────────────────── custmatch-server ──────────────────────────┐
  files ──► inbox│ scheduler ── load ──────────────┐                                     │
                 │     │                           ▼                                     │
                 │     └─ batch run ──publish──► ┌──────────── the database ───────────┐ │
                 │       (the custmatch          │ source records and every version,   │ │
                 │        library)               │ loads, runs (which one is current), │ │
  your apps ────►│ API: lookups, POST /match ◄──►│ customers, links, golden records,   │ │
                 │   matcher: one writer,        │ history, steward decisions,         │ │
                 │   index in memory             │ audit log, API keys                 │ │
  stewards ─────►│ dashboard ◄──────────────────►└─────────────────────────────────────┘ │
                 └───────────────────────────────────────────────────────────────────────┘

• One database holds everything shared: the source records and their loads, the runs and which one is current, the results, stewards' decisions and the matcher's lock. Postgres for production (many readers and writers, advisory locks, managed everywhere); SQLite on a persistent disk for a trial or one small team. The code targets both, so moving from trial to production is configuration.
• Loads keep history. A load is full (records missing from it are marked deleted, not removed) or incremental. The current version of every record is kept, and every earlier version, so "what did this record look like at load N" has an answer. Only the mapped fields are stored.
• The batch run is the truth. It is custmatch run over every live record - loaded from files or arrived through the API - with labels and stewards' decisions read from the database, and customer IDs carried over from the current run. It publishes in one transaction: readers see the old run or the new one, never a mix. The current run and one before it are kept.
• Real-time matching is provisional. POST /match scores a record against the current run's model in memory - a compact index of hashed blocking keys, built from the batch blocker's own key definition - and joins it to the customer it fits: it must match at least half of that customer's records, not just one. It merges a second customer only on a link of 0.9 or more. Exactly one process writes (a lease); lookups keep working without it. Stewards' decisions, trusted IDs, the twin guard and duplicate-free sources apply at once; the next batch run reconciles everything, and changes reach users as merges and splits in the history.
• The tables (custmatch_server/schema.py, created by versioned migrations):

Start it

A trial on your machine, with Postgres and the demo data:

docker compose -f server/docker-compose.yml up -d --build db api
docker compose -f server/docker-compose.yml run --rm api key create me --role admin   # shown once: keep it

Load the demo (or your own files, below) and run a first batch - one scheduler pass, done by hand:

for s in web app store; do cp examples/demo/$s.csv server/inbox/$s.full.csv; done
docker compose -f server/docker-compose.yml run --rm scheduler \
  schedule /demo/config.json --labels /demo/labels.csv --inbox /inbox --once

Then start the scheduler, which from now on loads what arrives in the inbox and runs the batch at 02:00 UTC (only one scheduler works at a time, so a --once pass while it runs does nothing and says so):

docker compose -f server/docker-compose.yml up -d scheduler

Then open http://localhost:8765/dashboard and paste a key with the steward (or admin) role.

The image holds the library's default model only. For the opt-in "matcher_model": "forest_catboost" (docs/report/03-which-matcher.md "Which classifier"), build it with CatBoost: CUSTMATCH_EXTRAS=catboost docker compose -f server/docker-compose.yml build, or docker build --build-arg CUSTMATCH_EXTRAS=catboost .... A run trained with it cannot be served by an image without it; the API says so.

For production, run the same image three ways (API, scheduler, and Postgres you already manage) on whatever runs containers for you, with your config and labels mounted where the commands point. Without Docker: pip install ./ ./server from the repository.

Configuration

The matching itself is configured by the custmatch config file (sources, columns, rules) and the labels file: see the guide. The database creates and upgrades its own tables when any process starts.

API keys

custmatch-server key create NAME --role ROLE   # prints the key once; only its hash is stored
custmatch-server key list
custmatch-server key revoke NAME               # stops working within 30 seconds

Give each person and each system its own key: a steward's decisions are recorded under their key's name. server/sql/roles.sql sketches least-privilege database roles for Postgres.

Getting records in

The inbox (with the scheduler running): drop a file named <source>.<mode>.<csv|parquet> - web.full.csv, store.incremental.2026-10-05.parquet - where source is a source in your config and mode is:

• full: the source's complete contents. Records missing from it are marked deleted (kept, with their history, but left out of matching).
• incremental: new and changed records only.

The scheduler waits until a file's size has stopped changing, loads it, and moves it to inbox/done/ (or inbox/failed/, with an alert). The image runs as user 10001: on Linux, make the inbox folder writable by it (chown 10001 inbox). Each load is recorded: what it inserted, updated, deleted and left unchanged, and every version of every record.

By hand: custmatch-server load CONFIG SOURCE FILE [--full].

Through the API: records matched with POST /match and "write": true are stored as that day's API load and included in the next batch run.

Batch runs and the schedule

custmatch-server schedule CONFIG --labels LABELS --inbox DIR --batch-at 02:00 loads the inbox as files arrive and runs the batch at 02:00 UTC daily (repeat --batch-at for more). A batch run matches every live record, keeps customer IDs stable from the previous run, publishes in one transaction (readers see the old run or the new one, never a mix), keeps the previous run for rollback, and runs the health checks. Only one scheduler works at a time, even if two are started.

With your own scheduler (cron, a Kubernetes CronJob, Airflow), run custmatch-server schedule ... --once instead: one pass - load the inbox, run a batch, check - exiting 0 (ok), 1 (warnings) or 2 (failed).

Stewards' decisions train the model. Each "same" or "different" a steward records overrides that pair at once, and the next batch run also trains on it. Measured on the demo (server/learning_test.py, research/results/server_learning.json): starting from 200 random labels, three rounds of answering the review queue lifted clustered F1 from 79.5 to 88.8 when decisions train the model, against 84.1 when they only override their own pairs.

The dashboard

/dashboard, with a steward key:

• Overview: health, the current run, loads, recent decisions.
• Model & data: the threshold and accuracy estimates, score distribution, what the model relies on, completeness and common values by source, records too sparse to link, blocking, runs compared.
• Customers: search for a person; browse customers worth a second look (largest, weakest link, ...); sample links by score. A customer shows its golden record, its records as a network of links, and why each link scored what it did.
• Review: the uncertain pairs, most uncertain first: same person, different people, or skip (keys S, D, K). Filter by source, the model's call or score; a list mode decides many at once; a note goes with each decision. Pairs shown to one steward are held from the others for 15 minutes.
• Decisions: every decision with its author and note; revoke one, approve or reject proposals (with CUSTMATCH_FOUR_EYES=on), download them as CSV; steward activity, including how often each steward overturned the model.

In a customer: golden-record lineage (which records hold each value) and pinning a value, which beats survivorship in this and every later run; merging with another customer and splitting a record out, both previewed before they happen and applied at once; the household; every steward action on its records. Under Customers, Compare puts any two records side by side and says whether blocking would ever compare them (why they did not match); Find possible matches scores a person you type in, without saving anything. Every panel has an (i) explaining what it shows, how it is calculated and what good looks like.

Monitoring

GET /health (no key) says the process is up. GET /status and custmatch-server check run the health checks; the scheduler runs them after every batch and posts anything not ok to CUSTMATCH_ALERT_WEBHOOK.

Backup and restore

Back up two things together: the database and the data directory.

pg_dump -Fc "$DATABASE" > custmatch-$(date +%F).dump        # Postgres
tar czf custmatch-runs-$(date +%F).tgz -C "$CUSTMATCH_DATA_DIR" runs

With SQLite, stop the processes and copy the .db file (or use sqlite3 custmatch.db ".backup copy.db"). Restore both from the same time: the database names the run directories it uses. Only the current run and the one before it are kept, so older run directories can be deleted or archived.

Upgrading

1. Back up (above).
2. Stop the scheduler, so no batch run is in progress.
3. Start the new image's processes: the database upgrades itself at start (migrations).
4. Run custmatch-server check.

Migrations do not run backwards automatically; to go back, restore the backup and the old image.

Sizing

Measured with Postgres on a laptop (server/scale_test.py, research/results/server_scale.json), per million records: loads about 1 minute, a batch run about 2.5 minutes (2-3 GB of memory), the real-time matcher starts in 6 seconds and holds about 0.5 GB, a real-time match takes 24 ms (median) and a lookup 7 ms; the database grows by about 1.8 GB. The batch library has run 10 million records on one 16-core machine (docs/report/06-blocking-and-scale.md, "Larger files").

Troubleshooting