dupehell 0.4.3

Synthetic data generator for record linkage benchmarking.
Rust + Python — 40 domains, 500K+ rec/s.

Generate synthetic multi-entity datasets with realistic schemas, controlled duplicates, hard negatives, and ground-truth labels. Designed for benchmarking entity resolution (deduplication) and record linkage pipelines.

Quick start

Python (pip)

pip install dupehell

from dupehell import generate

r = generate(domain="publishing", size=10000, seed=42, difficulty="hard")
print(r.dataset)       # ./publishing_<hash>.ipc
print(r.ground_truth)  # ./publishing_<hash>_ground_truth.ipc
print(r.total_records) # ~10150

CLI (Rust)

cargo run --release -- --domain kyc --size 100000 --seed 42

Output

Each run produces:

• {domain}_{hash}.ipc — main dataset
• {domain}_{hash}_ground_truth.ipc — ground-truth labels

CLI options

Features

• 40 domains — KYC, publishing, fintech, blockchain, technology, banking, healthcare, ecommerce, automotive, cybersecurity, gaming, and 30 more
• Multi-entity schemas — 3–5 entity types per domain (person, account, address, transaction)
• Controlled noise — typos, OCR errors, homoglyphs, date swaps, phonetic variants, Unicode pollution
• Hard negatives — same_field, mix_identifier, mix_arrays primitives
• Ground truth — full match labels (exact_dup, hard_neg, singleton) with cluster statistics
• Deterministic — seeded RNG (rand_pcg) for reproducible output

Performance

All runs on Lenovo ThinkPad P16 Gen 2 — Intel Core i7 13th, 32 GB DDR5, SK Hynix 1 TB NVMe. Difficulty hell, IPC format. Throughput averaged across all 40 domains.

Multi-domain throughput (hell, IPC)

IPC vs Parquet

Difficulty hell, domain-average throughput.

See docs/BENCHMARK.md for KYC medium-difficulty single-domain metrics and full per-domain breakdowns at all sizes.

Architecture

lib.rs / main.rs → Context (133 pools) → PipelineConfig → run_pipeline()
                                                          │
         ┌────────────────────────────────────────────────┼────────────────────┐
         ▼                                                ▼                    ▼
  entity_gen.rs                                    fk_remap.rs           hn_common.rs
  (batch gen)                                      (FK cross-ref)        (hard negatives)
         │                                                │                    │
         └────────────────────────────────────────────────┴────────────────────┘
                                                          ▼
                                                     pipeline.rs
                                               (merge + GT + IPC write)
                                                          ▼
                                               {domain}.ipc + GT.ipc

Documentation

Domains

Academia · Agriculture · Automotive · Aviation · Banking · Biotech · Blockchain · Construction · CRM · Cybersecurity · Ecommerce · Education · Energy · Fashion · Fintech · Food & Beverage · Gaming · Healthcare · Hospitality · HR · Insurance · KYC · Legal · Logistics · Manufacturing · Maritime · Media · Mining · Nonprofit · Pharma · Publishing · Real Estate · Renewable Energy · Retail · Social Media · Sports · Supply Chain · Technology · Telecom · Travel

Roadmap

• Graph generation — model entity relationships as property graphs (nodes, edges, attributes) for graph-based entity resolution and community detection benchmarking
• Synthetic identity module — generate realistic digital identities (browser fingerprints, device profiles, network patterns) for cybersecurity simulation and threat detection research
• Performance — continue pushing throughput via smarter batching, column-level parallelism, and reduced allocations

Development

cargo test        # 113 tests, ~30s
cargo build --release
cargo clippy      # 0 warnings
cargo fmt --check # all formatted

Python wheel

pip install maturin
maturin build --release
pip install target/wheels/dupehell-*.whl

License

MIT — Educational Use Only.

This software generates synthetic data for research and educational purposes only. It must not be used for fraud, identity theft, surveillance, or any illegal activity. See ETHICS.md for the full list of prohibited uses and responsible disclosure policy.

If you use DupeHell in your research, please cite:

@software{dupehell2026,
  author = {DupeHell Contributors},
  title = {DupeHell: Synthetic Multi-Domain Dataset Generator for
           Record Linkage Benchmarking},
  year = {2026},
  url = {https://github.com/vntoinekaio/DupeHell}
}