dupehell 0.5.4

Synthetic data generator for record linkage benchmarking.
Rust + Python — 40 domains, 500K+ rec/s.

Generate synthetic multi-entity datasets with realistic schemas, controlled duplicates, hard negatives, and ground-truth labels. Designed for benchmarking entity resolution (deduplication) and record linkage pipelines.

Quick start

Python (pip)

pip install dupehell

from dupehell import generate

r = generate(domain="publishing", size=10000, seed=42, difficulty="hell")
print(r.dataset)       # ./publishing_<hash>.parquet
print(r.ground_truth)  # ./publishing_<hash>_ground_truth.parquet
print(r.total_records) # ~10150

CLI (Rust)

cargo run --release -- --domain kyc --size 100000 --seed 42

Output

Each run produces:

• {domain}_{hash}.parquet — main dataset
• {domain}_{hash}_ground_truth.parquet — ground-truth labels
• {domain}_{hash}_nodes.parquet / {domain}_{hash}_edges.parquet — property graph (only with --graph / generate_graph=True, see below)

CLI options

Graph generation

Add --graph (CLI) or generate_graph=True (Python) to additionally emit a property graph alongside the usual tabular dataset — nodes (one per record, same attributes as the dataset) and typed edges (fk, exact_dup, fuzzy_dup, hard_neg) linking record_ids that a record-linkage/graph pipeline should resolve to the same entity or a hard negative. Disabled by default: tabular output, RNG sequence, and memory footprint are unchanged when omitted.

r = generate(domain="fintech", size=10000, seed=42, generate_graph=True)
print(r.nodes)  # ./fintech_<hash>_nodes.parquet
print(r.edges)  # ./fintech_<hash>_edges.parquet

cargo run --release -- --domain fintech --size 10000 --seed 42 --graph

Features

• 40 domains — KYC, publishing, fintech, blockchain, technology, banking, healthcare, ecommerce, automotive, cybersecurity, gaming, and 30 more
• Multi-entity schemas — 3–5 entity types per domain (person, account, address, transaction)
• Controlled noise — typos, OCR errors, homoglyphs, date swaps, phonetic variants, Unicode pollution
• Hard negatives — same_field, mix_identifier, mix_arrays primitives
• Ground truth — full match labels (exact_dup, hard_neg, singleton) with cluster statistics
• Graph generation — optional property graph output (nodes, typed edges) for graph-based entity resolution and community detection benchmarking
• Deterministic — seeded RNG (rand_pcg) for reproducible output

Performance

All runs on Lenovo ThinkPad P16 Gen 2 — Intel Core i7 13th, 32 GB DDR5, SK Hynix 1 TB NVMe. Difficulty hell, IPC format. Throughput averaged across all 40 domains.

Multi-domain throughput (hell, IPC)

Refreshed against the current pipeline, all 40 domains, single consistent sweep (see docs/BENCHMARK.md for the full per-domain table). kyc is a persistent outlier at every scale — its natural_person entity is the widest, most PII-dense schema in the crate, and at hell all 9 noise categories are simultaneously active on it (vs. often a single active category on sparser schemas). Excluding kyc, the 1M average is 705,333 rec/s. Throughput drops noticeably at 50M for the heavier/denser schemas (crm, ecommerce, insurance, technology, supplychain, telecom) — a known non-monotonic slowdown at higher record counts, tracked separately.

IPC vs Parquet

Difficulty hell, domain-average throughput across all 40 domains. Parquet is the CLI default (ZSTD(3) compressed); IPC is the internal/streaming-oriented format.

Parquet's cost shrinks as scale grows — the fixed per-file ZSTD compression overhead gets amortized over more rows. Parquet is a reasonable default at any scale tested here; IPC remains the better choice for pure raw-throughput or internal streaming use cases.

See docs/BENCHMARK.md for the full per-domain matrix (all 3 difficulties, --graph overhead, FK edge weighting) and single-domain metrics at scale.

Architecture

lib.rs / main.rs → Context (151 pools) → PipelineConfig → run_pipeline()
                                                          │
         ┌────────────────────────────────────────────────┼────────────────────┐
         ▼                                                ▼                    ▼
  entity_gen.rs                                    fk_remap.rs           hn_common.rs
  (batch gen)                                      (FK cross-ref)        (hard negatives)
         │                                                │                    │
         └────────────────────────────────────────────────┴────────────────────┘
                                                          ▼
                                                     pipeline.rs
                                               (merge + GT + IPC write)
                                                          ▼
                                               {domain}.ipc + GT.ipc

Documentation

Domains

Academia · Agriculture · Automotive · Aviation · Banking · Biotech · Blockchain · Construction · CRM · Cybersecurity · Ecommerce · Education · Energy · Fashion · Fintech · Food & Beverage · Gaming · Healthcare · Hospitality · HR · Insurance · KYC · Legal · Logistics · Manufacturing · Maritime · Media · Mining · Nonprofit · Pharma · Publishing · Real Estate · Renewable Energy · Retail · Social Media · Sports · Supply Chain · Technology · Telecom · Travel

Development

cargo test        # 113 tests, ~30s
cargo build --release
cargo clippy      # 0 warnings
cargo fmt --check # all formatted

Python wheel

pip install maturin
maturin build --release
pip install target/wheels/dupehell-*.whl

License

MIT.

This software is intended for research and educational use: generating synthetic data for record linkage / entity resolution benchmarking. It must not be used for fraud, identity theft, surveillance, or any illegal activity. See ETHICS.md for the full list of prohibited uses and responsible disclosure policy.

If you use DupeHell in your research, please cite:

@software{dupehell2026,
  author = {DupeHell Contributors},
  title = {DupeHell: Synthetic Multi-Domain Dataset Generator for
           Record Linkage Benchmarking},
  year = {2026},
  url = {https://github.com/vntoinekaio/DupeHell}
}