deidkit 0.1.3

Schema-driven pseudonymization for clinical/tabular datasets: synthetic names, interval-preserving date shifting, multi-stage free-text PII detection, and before/after audit exports.

deidkit

Schema-driven de-identification for clinical & tabular data. Point it at your tables; get back a usable, pseudonymized copy plus a full audit of every change.

It replaces names with realistic fakes (in columns and inside free text), shifts dates while preserving every interval, swaps IDs/phones/emails for stable surrogates, and sends anything uncertain to a human-review sheet instead of guessing. It is deterministic (same input + same secret ⇒ same output) and re-runnable.

---

Contents

1. Install
2. Quick start
3. Inputs — what you feed it
4. What it does to each column
5. Outputs — what you get back
6. Intermediate outputs — inspect before you trust
7. Training it — the review → learn loop
8. Tuning recall & precision
9. Reading from a database (Fabric / Azure SQL)
10. Database connection parameters — exactly what to fill in
11. What to look out for
12. Full option reference (CLI + Python)
13. Troubleshooting

---

1. Install

pip install deidkit

One system-level thing is required once, on the machine that will connect: the Microsoft ODBC Driver 18 for SQL Server — the actual database driver that pyodbc talks to:

# macOS
brew tap microsoft/mssql-release https://github.com/microsoft/homebrew-mssql-release
brew install msodbcsql18
# Linux (Debian/Ubuntu): install Microsoft's "msodbcsql18" package
# Windows: install the "ODBC Driver 18 for SQL Server" MSI

Requires Python 3.9+. Confirm it:

deidkit --version

---

2. Quick start

Your data is the Azure SQL Database mirrored into Microsoft Fabric, so the run connects to that endpoint, de-identifies the six tables together, and writes a clean copy out. That's the whole thing:

deidkit init-secret --out secret.key       # do this once; keep the file private

import deidkit as dk
from deidkit.io import read_sql_table

# connection string built from the values in section 10 (nothing writes back)
CONN = (
    "mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>"
    "?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive"
)
TABLES = ["tbl_patient", "tbl_encounter", "tbl_notes",
          "tbl_lab", "tbl_imaging", "tbl_medication"]

tables = {t: read_sql_table(CONN, t) for t in TABLES}     # read the DB (read-only)
result = dk.deidentify(
    tables, out="deid_out",                               # de-identify all six together
    secret_file="secret.key",
    lang="es",
    entity_key="patient_id",
    known_id_columns=["patient_id"],
)
print(result.summary)
# -> deid_out/ (ship this)  +  deid_out-PRIVATE/ (keep private — see section 5)

That reads the tables from Fabric, never writes back, and produces the two output folders. Section 9 walks through the connection (plus a database-to-database variant), and section 10 lists the exact connection values to put in CONN.

Handed a file export instead of database access? If you receive the tables as files (CSV etc.) rather than a live endpoint, the same run is one CLI line — deidkit run data/ — writing data_deid/ and data_deid-PRIVATE/. See section 3.

CLI or code — same engine. Everything below is shown for both interfaces. The rule of thumb: every CLI --flag is a field on the Python Policy object (e.g. --mode balanced ⇄ Policy(mode="balanced"), --ignore blood_type ⇄ Policy(ignore={"blood_type"})). Pick whichever fits your workflow — a one-off delivery (CLI) or a pipeline/notebook (code).

---

3. Inputs — what you feed it

Your source: the Fabric / Azure SQL database

For this project the input is your live database — the Azure SQL Database mirrored into Microsoft Fabric. deidkit connects to the Fabric SQL analytics endpoint (read-only), reads the six tables directly, de-identifies them, and writes the clean copy out. It never writes back to your database.

deidkit connects on its own — it only needs the connection details for that endpoint (the SQL analytics endpoint host, the database name, and how to authenticate). Those go into one connection string; section 9 shows how a run works and section 10 lists the exact values. Start there.

Or: a data export as files

If instead you hand over a data extract as files (one table per file in a folder), deidkit reads those too — .csv, .tsv, or .json. One file = one table, and the filename becomes the table name. CSVs are read as text (so IDs and date formats aren't mangled) and tolerate non-UTF-8 exports.

deidkit run data/            # a folder of table files

Two things that matter about your columns

• patient_id is the anchor. By default entity_key = patient_id. Every date for a patient is shifted by the same offset, and every real value maps to the same surrogate — across all tables — so joins and time-intervals survive. If your patient column has a different name, set --entity-key <col>. All six Meridian tables share patient_id, so this works out of the box.
• Column names drive the plan. deidkit recognises clinical column names in English and Spanish (e.g. note_full, result_interpretation, report_text, clinical_justification_pbs) and classifies each one automatically. You can override any decision (section 12). An optional data-dictionary JSON (--schema dict.json) makes the classification authoritative instead of heuristic.

---

4. What it does to each column

Every column gets exactly one strategy. Most are chosen automatically; you can force any of them with a rule.

strategy	effect	typical columns
passthrough	left untouched (the safe default)	sex, hemoglobin, diagnosis, insurer
date_shift	move by the per-patient offset — intervals preserved	note_datetime, encounter_start/end, birth_date
freetext	detector scrubs names/IDs/phones/emails/dates inside the text	note_full, report_text, result_interpretation
synthetic_name	a name value → realistic fake name	a patient_name / provider_name column
identifier	an ID value → stable shape-preserving surrogate	a raw cédula / document number
age_band	bucket an age (e.g. 10-yr bands, 90+)	age_years
generalize_year	reduce a date to its year
redact	blank the value
drop	remove the column entirely

Preview the full plan before running anything with deidkit plan (section 6).

---

5. Outputs — what you get back

A run always produces two folders, kept separate on purpose so that zipping the deliverable can never leak the key:

deid_out/                         ← SHIP THIS — de-identified data only
    tbl_patient.csv
    tbl_encounter.csv
    tbl_notes.csv
    tbl_lab.csv
    tbl_imaging.csv
    tbl_medication.csv

deid_out-PRIVATE/                 ← NEVER SHARE — the re-identification key
    deidkit-secret.key            ← the secret (only if it was auto-generated)
    mapping.private.json          ← value → surrogate map (reverses/extends the run)
    deid_audit.PRIVATE.xlsx       ← before/after audit workbook (see section 6)
    deid_audit.PRIVATE_freetext_changes.csv
    review.csv                    ← uncertain detections for a human (see section 7)

• deid_out/ — the pseudonymized tables, same structure as the input. This is the only thing you deliver.
• deid_out-PRIVATE/ — the secret + mapping + audit. Whoever holds these can re-identify. Store them somewhere else and never include them in a delivery.

Output format defaults to CSV; use --format parquet|tsv|json (CLI) or fmt= (Python) to change it. Reading from a DB and want the result written back to a DB? See run-db in section 9.

---

6. Intermediate outputs — inspect before you trust

Don't run blind. deidkit gives you three ways to look between input and final output.

deidkit plan — the decision for every column (changes nothing)

deidkit plan data/

table            column                           strategy         decided_by
----------------------------------------------------------------------------------------
tbl_notes        note_full                        freetext         heuristic:freetext  *
tbl_lab          result_interpretation            freetext         heuristic:freetext  *
tbl_lab          order_datetime                   date_shift       heuristic:date  *
tbl_patient      sex                              passthrough      default

Columns marked * will change. If something is wrong, fix it with --ignore, --only, --name-column, or a rule, then re-run plan until it's right.

In code: deid.plan_dataframe(df, "tbl_notes") returns the same {column: PlanItem} decision for one table.

deidkit scan — what the free-text detector sees on a snippet

deidkit scan --text "Paciente Juan Pérez, CC 1.032.456.789, tel [phone]"

accepted  type         conf  detectors              text
--------------------------------------------------------------------------------
REPLACE   PERSON       0.99  context+gazetteer      'Juan Pérez'
REPLACE   NATIONAL_ID  0.99  regex                  '1.032.456.789'
REPLACE   PHONE        0.99  regex                  '300 123 4567'

REPLACE = will be scrubbed. review = queued for a human instead.

In code: dk.Detector("es", "conservative").detect(text) returns the same detections.

The audit workbook — evidence of every change

deid_out-PRIVATE/deid_audit.PRIVATE.xlsx has:

sheet	what it shows
Summary	counts by entity type and column
Field plan	the strategy chosen for every column, and why
Free-text changes	every before → after edit inside notes/reports, and which detector fired
Structured changes	before → after for structured columns (dates, IDs, names)
Mapping	the private value → surrogate map
Review queue	detections that were not auto-replaced

Verify intervals survived (they should, exactly):

import pandas as pd
enc = pd.read_csv("deid_out/tbl_encounter.csv")
los = pd.to_datetime(enc["encounter_end"]) - pd.to_datetime(enc["encounter_start"])
print(los.describe())   # identical to the source — only absolute dates moved

---

7. Training it — the review → learn loop

deidkit never guesses on an ambiguous name. Anything it detects but isn't sure about (for example a lone surname that is also a common word) is left untouched and listed in review.csv for a person to rule on. Confirm or reject those, and the next run does better — deterministically, with no model.

review.csv holds only that uncertain tail; the confident detections are already scrubbed in the output and never appear here. It is written to the private folder on every run (deid_out-PRIVATE/review.csv) and contains raw candidate text, so treat it as sensitive — never ship it.

What the reviewer does

Open deid_out-PRIVATE/review.csv (Excel, Google Sheets, any editor). Each row is one uncertain candidate, with a context snippet so you can judge it. Fill in the first column only — decision(y/n) — and save:

decision(y/n)	entity_type	candidate	context	table	column	row_id	confidence	detectors
y	PERSON	Bello	…paciente Bello refiere dolor…	tbl_notes	note_full	87	0.6	context
n	PERSON	Cruz	…la Cruz Roja atendió…	tbl_notes	note_full	42	0.55	gazetteer
(blank)	PERSON	Sur	…zona Sur del hospital…	tbl_notes	note_full	91	0.5	gazetteer

• y — yes, this is a real person's name (it should be redacted).
• n — no, it's a false positive — not a name (e.g. "Cruz" in "Cruz Roja").
• blank — undecided; that row is skipped and simply stays for a later pass.

That is the whole reviewer job: read the context, type y or n, save the file. (Lenient spellings work too: yes / 1 / true = yes; no / 0 / false / fp = no.)

The four steps

# 1. Run. Uncertain detections are written to deid_out-PRIVATE/review.csv.
deidkit run <input> deid_out

# 2. A reviewer opens deid_out-PRIVATE/review.csv and marks each row y / n (above).

# 3. Fold those decisions into a "learn" directory.
deidkit learn --review deid_out-PRIVATE/review.csv --learn-dir learned/

# 4. Re-run pointing at what it learned: confirmed names are now caught
#    automatically, and rejected candidates are suppressed.
deidkit run <input> deid_out --learn-dir learned/

Step 3 appends to two plain-text files that learned/ accumulates across cycles (so it keeps improving):

• learned_names.txt — words from every y → added to the name list (higher recall),
• learned_stoplist.txt — words from every n → added to the stoplist (higher precision).

Both are one token per line and human-editable. In code, load them with Policy(extra_name_files=["learned/learned_names.txt"], extra_stoplist_files=["learned/learned_stoplist.txt"]).

Good to know:

• review.csv is produced on every run — --learn-dir is only needed to load what was learned (steps 3–4), not to create the queue.
• Learning is token-level: marking "María Gómez" as y teaches both maría and gómez (lowercased, accent-folded), so they are caught elsewhere too.
• The loop tunes the borderline cases. A name the detector missed entirely will not appear in review.csv — for those, use the known-PHI columns below.

Two stronger levers (deterministic, near-100% recall)

• Known-PHI columns — if you already know a row's real name or ID (from a structured column), force-scrub those exact strings from that row's free text. No guessing:

  # CLI
  deidkit run data/ deid_out \
      --known-name-column patient_name \
      --known-id-column patient_id
  

  # code
  dk.Policy(known_name_columns=["patient_name"], known_id_columns=["patient_id"])
  

• Extend the name lists — feed your own gazetteer / stoplist:

  # CLI
  deidkit run data/ deid_out \
      --extra-name-file my_surnames.txt \
      --extra-stoplist-file never_redact.txt
  

  # code
  dk.Policy(extra_name_files=["my_surnames.txt"],
            extra_stoplist_files=["never_redact.txt"])
  

---

8. Tuning recall & precision

Lever	What it does
--mode conservative (default)	replace only clear names; queue the rest to review.csv
--mode balanced / aggressive	replace more eagerly (higher recall, more to review)
--spacy-model es_core_news_lg	add a statistical NER second opinion (needs deidkit[ner])
--id-checksums DNI,NIE	catch bare national IDs (no cue word) with near-zero false positives
--name-style tagged|placeholder|token|realistic	how fakes look — tagged = "Juan Gómez [SINTÉTICO]" (obvious); realistic = unmarked
--no-medical-vocab	(advanced) stop treating ~21k drug/disease terms as safe non-names

Start conservative, work the review queue down, and turn on the NER stage only if you need more recall — the audit is how you prove the result.

Enabling NER is a one-time model install (the model is ~500 MB and ships separately from the package, not bundled):

pip install "deidkit[ner]"                 # installs spaCy
deidkit download-model es_core_news_lg      # fetches the model (once)
deidkit run <input> deid_out --spacy-model es_core_news_lg

If the model isn't installed, deidkit still runs on the deterministic rules and prints a clear warning — it never fails, but free-text name recall is lower.

In code, the same knobs are Policy fields:

dk.Policy(mode="balanced", spacy_model="es_core_news_lg",
          id_checksums=["DNI", "NIE"], name_style="tagged")

---

9. Reading from a database (Fabric / Azure SQL)

How the connection works

deidkit talks only to the Fabric SQL analytics endpoint — it never logs into the Azure portal or the production database. The setup is the standard Fabric mirror:

1. The Azure SQL Database is mirrored into a Microsoft Fabric workspace, and the mirrored database is shared read-only — Read + Read all SQL analytics endpoint data, nothing else. No write or admin access is ever required.
2. deidkit is pointed at that endpoint with a connection string built from four values (see section 10): the SQL analytics endpoint, the database name, the authentication method, and the workspace URL.
3. It connects to the read-only endpoint, reads the six tables, de-identifies them, and writes out the clean copy. It never writes back — it only reads.

All that's needed is read-only access to the shared endpoint plus those values.

Then it runs — two patterns

The database drivers ship with deidkit (section 1). Pick based on where you want the output.

Pattern 1 — Database → files (recommended, simplest)

Read the tables from Fabric into memory, de-identify them together, write clean files. Python only:

import deidkit as dk
from deidkit.io import read_sql_table

CONN = (
    "mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>"
    "?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive"
)

TABLES = ["tbl_patient", "tbl_encounter", "tbl_notes",
          "tbl_lab", "tbl_imaging", "tbl_medication"]

# Read all tables so they share one map + one set of per-patient date offsets
tables = {t: read_sql_table(CONN, t) for t in TABLES}

result = dk.deidentify(
    tables,
    out="deid_out",
    secret_file="secret.key",
    lang="es",
    entity_key="patient_id",
    known_id_columns=["patient_id"],
)
print(result.summary)
# -> deid_out/*.csv  (ship)   +   deid_out-PRIVATE/  (keep)

Pattern 2 — Database → database (run-db / deidentify_database)

Write the de-identified result into a separate output database that mirrors the source structure (same tables, columns, types). You must provision the output DB yourself — deidkit refuses to run without it and never edits the source.

# CLI
deidkit run-db \
    --source "mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive" \
    --out-db "postgresql://deid_user:PASSWORD@your-host:5432/deid_db" \
    --policy examples/policy.meridian.yaml \
    --secret-file secret.key

# code
import deidkit as dk

dk.deidentify_database(
    source="mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>"
           "?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive",
    out_db="postgresql://deid_user:PASSWORD@your-host:5432/deid_db",
    secret_file="secret.key",
    lang="es",
    entity_key="patient_id",
    known_id_columns=["patient_id"],
)

• --out-db / out_db is required and must differ from the source.
• A column whose de-identified value no longer fits the source type (e.g. an age turned into a band) is widened to text; every other column keeps its structure.
• The secret + mapping + audit go to ./deidkit-db-PRIVATE/ (never the output DB).

---

10. Database connection parameters — exactly what to fill in

The four connection values

Once the mirrored database is shared read-only, gather these four values (they are exactly what the Fabric "share" step produces):

#	Item	Example	Used for
1	SQL Analytics Endpoint (the host)	abc123....datawarehouse.fabric.microsoft.com	connecting
2	Database name	ProductionMirror	connecting
3	Authentication method — Microsoft Entra ID or SQL Login	Microsoft Entra ID	connecting
4	Fabric Workspace URL	https://app.fabric.microsoft.com/groups/...	locating & confirming the shared database

Items 1–3 go into the connection string; item 4 (the workspace URL) only locates and confirms the shared database in Fabric — it is not part of the connection string.

The connection string

Connections are SQLAlchemy URLs. For the Fabric source, the shape is:

mssql+pyodbc://@<HOST>/<DATABASE>?driver=<DRIVER>&authentication=<AUTH>

Placeholder	Filled from	Example
<HOST>	item 1 — the SQL analytics endpoint	abc123....datawarehouse.fabric.microsoft.com
<DATABASE>	item 2 — the database name	ProductionMirror
<AUTH>	item 3 — the authentication method	ActiveDirectoryInteractive
<DRIVER>	installed locally (section 1), not a Fabric value	ODBC+Driver+18+for+SQL+Server

Note the @ before <HOST>: ...pyodbc://@abc123.... With Entra ID auth there is no username/password in the URL — the @ with an empty user is intentional. Spaces in the driver name are written as + (URL-encoded).

Authentication options for <AUTH>:

• Microsoft Entra ID, interactive (opens a browser to sign in) — best for a person running it manually:

  ...?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive
  

• Microsoft Entra ID, service principal (unattended jobs / servers):

  mssql+pyodbc://<CLIENT_ID>:<CLIENT_SECRET>@<HOST>/<DATABASE>?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryServicePrincipal
  

• SQL login (only if they gave you a SQL username/password instead of Entra):

  mssql+pyodbc://<USER>:<PASSWORD>@<HOST>/<DATABASE>?driver=ODBC+Driver+18+for+SQL+Server
  

Install the ODBC driver (<DRIVER> above must match what’s installed):

# macOS
brew tap microsoft/mssql-release https://github.com/microsoft/homebrew-mssql-release
brew install msodbcsql18

# Debian/Ubuntu — follow Microsoft’s "ODBC Driver 18 for SQL Server" apt instructions
# Windows — install the "ODBC Driver 18 for SQL Server" MSI

The output DB URL (Pattern 2 only) is any SQLAlchemy URL for a database you created and can write to:

Target	URL shape
PostgreSQL	postgresql://user:password@host:5432/deid_db
Azure SQL / SQL Server	mssql+pyodbc://user:password@host/deid_db?driver=ODBC+Driver+18+for+SQL+Server
Local SQLite (quick test)	sqlite:///deid_local.db

Quick connectivity check before a full run:

from sqlalchemy import create_engine, text
CONN = "mssql+pyodbc://@<HOST>/<DATABASE>?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive"
with create_engine(CONN).connect() as c:
    print(c.execute(text("SELECT TOP 1 patient_id FROM tbl_patient")).fetchone())

---

11. What to look out for

• Is patient_id a raw ID or already hashed? By default it's passed through (it's the join key). If it's actually the cédula / national ID, that's a leak — add FieldRule("patient_id", "identifier") (or uncomment the rule in examples/policy.meridian.yaml). Surrogating it still preserves joins.
• Filenames & paths can hide PHI. Columns like result_file_name and image_relative_uri often embed a name or MRN. They pass through by default — redact them if you don't need them.
• Keep the secret and mapping private. They are the re-identification key. Never put *-PRIVATE/ in a delivery. Losing the secret means you can't re-run consistently; leaking it undoes the de-identification.
• Determinism needs the same secret. Pass the same --secret-file every time, or every run produces different surrogates.
• No detector is perfect. Pseudonymization reduces but does not eliminate re-identification risk (per NIST SP 800-188 / HIPAA expert-determination). Review the audit, consider quasi-identifier combinations (age + stratum + location …), and treat outputs accordingly.

---

12. Full option reference (CLI + Python)

CLI commands

Command	Purpose
deidkit init-secret --out secret.key	generate a keying secret (chmod 600)
deidkit plan <input>	show the strategy for every column; change nothing
deidkit run <input> [out]	de-identify files → files
deidkit run-db --source <url> --out-db <url>	de-identify a DB → a separate DB
deidkit scan --text "…"	show detections on a snippet (tuning/debug)
deidkit learn --review review.csv --learn-dir learned/	fold review decisions into learned dicts
deidkit map --mapping mapping.private.json	dump the private value→surrogate map

Run deidkit <command> --help for any command's flags.

CLI flags shared by run, run-db, plan, scan

Flag	Meaning
--policy FILE	YAML policy file; CLI flags override it
--lang {es,en,multi}	language for names/detection (multi = ES+EN)
--mode {conservative,balanced,aggressive}	free-text detection aggressiveness
--name-style {tagged,placeholder,token,realistic}	how fake names look (default tagged = obviously synthetic)
--entity-key COL	column that groups date shifts (default patient_id)
--date-max-days N	date-shift window in days (default 365; intervals preserved regardless)
--ignore a,b,c	never transform these columns
--only a,b	transform only these columns
--name-column COL	force a column to synthetic-name treatment (repeatable)
--known-name-column COL	column holding a known name to force-scrub from that row’s text (repeatable)
--known-id-column COL	column holding a known ID to force-scrub from that row’s text (repeatable)
--known-name "NAME"	a known name (e.g. clinic staff) scrubbed from all text (repeatable)
--id-checksums LIST	checksum-validated bare-ID detection: DNI,NIE (ES), CPF,CNPJ,CNS (BR), NHS (UK), CURP,RFC (MX), SSN, or ALL
--spacy-model NAME	enable the spaCy NER stage (e.g. es_core_news_lg)
--names-token	use [PERSON_x] tokens instead of realistic fake names
--no-date-text	do not shift date strings found inside free text
--no-medical-vocab	do not load the bundled medical safe-vocabulary
--extra-name-file FILE / --extra-stoplist-file FILE	merge extra gazetteer / stoplist files (repeatable)
--learn-dir DIR	auto-load learned_names.txt + learned_stoplist.txt from a previous learn

run-specific flags

Flag	Meaning
input / out (positional)	input path/glob/dir, and output dir (default <input>_deid)
--secret-file FILE / --secret STR	the keying secret (or set DEIDKIT_SECRET)
--schema FILE	data-dictionary JSON (auto-detected from the input folder if omitted)
--mapping FILE	private mapping JSON (default <out>-PRIVATE/mapping.private.json)
--report FILE	audit workbook path (default <out>-PRIVATE/deid_audit.PRIVATE.xlsx)
--format {csv,parquet,tsv,json}	output format (default csv)

run-db-specific flags

Flag	Meaning
--source URL	source DB SQLAlchemy URL (read-only) — required
--out-db URL	output DB SQLAlchemy URL you created — required, must differ from source
--tables a,b,c	limit to these tables (default: all)
--out-schema NAME	schema/namespace in the output DB
--if-exists {fail,replace,append}	behaviour if an output table exists (default fail)
--private-dir DIR	where secret+mapping+audit go (default ./deidkit-db-PRIVATE)

Python Policy fields

entity_key (default "patient_id"), date_max_days (365), lang ("es"), mode ("conservative"), name_style ("tagged"), shift_dates_in_text (True), only, ignore, rules (list[FieldRule]), spacy_model, known_name_columns, known_id_columns, extra_known_names, known_token_min_len (5), id_checksums, use_medical_vocab (True), extra_name_files, extra_stoplist_files.

FieldRule(column, strategy, table=None, options={}) — column and table accept * globs; the first matching rule wins.

Python functions

import deidkit as dk

# one call: files / DataFrame / dict of DataFrames -> Result
dk.deidentify(data, out=None, *, policy=None, schema=None, secret=None,
              secret_file=None, mapping=None, report=None, private_dir=None,
              fmt=None, table="table", **policy_kwargs)

# one DataFrame in, one clean DataFrame out
dk.deidentify_dataframe(df, table="table", **kwargs)

# database -> a separate database (mirrors source structure)
dk.deidentify_database(source, out_db, *, tables=None, schema=None,
                       out_schema=None, if_exists="fail", policy=None,
                       secret=None, secret_file=None, mapping=None,
                       private_dir=None, report=None, **policy_kwargs)

deidentify(...) returns a Result with .tables, .table (the single DataFrame when one went in), .summary, .out, .report_path, .mapping_path, .private_dir, and .secret (set only when a secret was auto-generated — save it to stay reproducible).

Full control — drive many tables through one shared map + one set of per-patient date offsets, and write the audit yourself:

import deidkit as dk

policy = dk.Policy(lang="es", mode="conservative", entity_key="patient_id",
                   known_id_columns=["patient_id"], ignore={"blood_type"})

deid = dk.Deidentifier(
    policy,
    secret_file="secret.key",                              # or secret="…", or $DEIDKIT_SECRET
    mapping_path="deid_out-PRIVATE/mapping.private.json",  # load+update for consistency
    # dict_index=dk.load_dictionary("dictionary.json"),   # optional data dictionary
)

tables = dk.io.read_tables("data/")          # {name: DataFrame}
clean = deid.run_dataset(tables)             # {name: DataFrame}, all sharing state
# or a single table:  deid.run_table(df, table="tbl_notes")

deid.save_mapping()
deid.write_report("deid_out-PRIVATE/deid_audit.PRIVATE.xlsx")
deid.write_review("deid_out-PRIVATE/review.csv")
print(deid.summary())

---

13. Troubleshooting

Symptom	Fix
ModuleNotFoundError: pyodbc / SQLAlchemy	Reinstall: pip install --force-reinstall deidkit (the DB drivers ship with it)
Data source name not found / driver error	Install ODBC Driver 18 for SQL Server and make the driver= value match exactly (section 10)
Browser never opens for ActiveDirectoryInteractive	Run from an interactive session; for servers use ActiveDirectoryServicePrincipal
cannot reach the output database (run-db)	Create the empty output DB first and pass a reachable --out-db URL
A column was changed that shouldn’t be	Add it to --ignore, or add a FieldRule(..., "passthrough"); confirm with deidkit plan
A free-text name slipped through	Try --mode balanced, enable --spacy-model, or add the name via the review→learn loop (section 7)
Non-UTF-8 CSV export	Handled automatically (deidkit falls back to latin-1/cp1252)
Different output each run	You’re not pinning the secret — pass the same --secret-file every time (section 2)

---

License

MIT.