FraudTwin

Documentation: main site

A deterministic payment world for building, breaking, and validating fraud systems.

FraudTwin is a Python framework for generating realistic financial behavior and using it to test fraud detection systems before production data or production infrastructure is available. It models entities, payment lifecycles, fraud campaigns, delayed labels, data-quality incidents, graphs, streaming events, and operational storage as one coherent system.

Why FraudTwin

Most synthetic-data tools generate independent rows. Most Kafka test fixtures exercise transport without domain truth. FraudTwin connects the two: every record has a stable identity, event time, causal context, and an auditable relationship to the source simulation.

The important boundary is explicit: observable data contains what a detector could know at a chosen time; oracle data contains the complete explanation used for evaluation and audit. This makes leakage and label-delay mistakes visible instead of silently rewarding them.

Quick start

Requirements: Python 3.12+. Install the package in an existing project with Poetry, or install it directly into a virtual environment with pip:

# Poetry-managed project
poetry add fraudtwin
poetry run fraudtwin --help

# Generate and persist the built-in minimal local run.
poetry run python -c 'import fraudtwin; run = fraudtwin.generate(write=True, output_dir="runs"); print(f"Run generated: {run.run_id}")'

# Or a virtual environment managed with pip
python -m pip install fraudtwin
fraudtwin --help
python -c 'import fraudtwin; run = fraudtwin.generate(write=True, output_dir="runs"); print(f"Run generated: {run.run_id}")'

The built-in minimal configuration is validated during generation and writes an immutable run manifest plus Parquet tables under runs/<run-id>/. For custom YAML configurations and repository workflows, see the installation and support matrix.

Python API

The public API is typed and has two explicit workflows:

from pathlib import Path

import fraudtwin

# In-memory: precise IDE autocomplete for entities, behavior, and datasets.
data = fraudtwin.generate()
dataset = data.require_dataset().frame
print(data.run_id, dataset.shape)

# Persisted: metadata and paths first; load records only when needed.
run = fraudtwin.generate(
    write=True,
    output_dir=Path("runs"),
)
loaded = run.load_data()
print(run.run_id, len(loaded.behavior.payments), run.manifest_path)

Use GeneratedData.require_dataset() when the configuration enables a point-in-time dataset. Use GeneratedRun.load_data() when a persisted run is needed by a graph, replay, dataset, or publisher workflow. Both methods keep the contract explicit and make class members discoverable to Pylance, mypy, and other Python language servers.

From simulation to system test

FraudTwin is intended to be used as a progression, not a single generator call:

configuration
    → entities and behavior
    → payment/lifecycle events
    → fraud campaigns and delayed labels
    → point-in-time datasets and graphs
    → models, backtests, and drift reports
    → Kafka/PostgreSQL/Iceberg/serving integration tests

Every stage can run offline. External services add realism but do not change the source truth or block the core learning path.

Learning paths

The documentation is organized around engineering tasks rather than a flat notebook directory:

The versioned documentation site contains rendered notebooks, guides, troubleshooting, compatibility notes, and the Python API reference. The API reference lists supported public classes and functions with signatures, parameters, return types, exceptions, and source links.

For a complete workflow, see the ML evaluation methodology and the integration runbooks.

What is modeled

• Customers, institutions, accounts, cards, merchants, devices, and payment habits.
• CARD, PIX-like, and account-transfer lifecycles with ledger invariants.
• Fraud scenarios, campaigns, camouflage, difficulty, hard negatives, alerts, cases, disputes, and delayed labels.
• Observable and oracle views with label-observation histories.
• Point-in-time datasets, replay, rolling backtests, baseline evaluation, calibration, counterfactuals, and drift reports.
• Graph nodes, edges, campaigns, evidence, Neo4j exports, and PyTorch Geometric conversion.
• Deterministic quality faults, schema evolution, Kafka chaos, partitioning, checkpoints, and reconciliation.
• A versioned extension SDK for custom fraud scenarios, payment rails, behavior models, fault injectors, and output sinks.

Generated outputs

Each persisted run is self-describing:

runs/<run-id>/
├── manifest.json
├── entities/*.parquet
├── behavior/ behavior_profiles.parquet
├── payments/ payments.parquet payment_events.parquet
├── ledger/ ledger_entries.parquet
├── fraud/ fraud_records.parquet fraud_alerts.parquet fraud_labels.parquet
└── ml/ dataset.parquet dataset_manifest.json

The manifest records the seed, validated configuration, schema versions, row counts, fingerprints, and quality diagnostics. Graph, benchmark, replay, counterfactual, and campaign workflows add their own manifests without rewriting the original source records. No real personal data or payment credentials are generated.

Optional integrations

Install only what a workflow needs. The base install remains dependency-light.

# Poetry-managed project
poetry add fraudtwin --extras ml --extras graph
poetry add fraudtwin --extras kafka --extras postgres --extras lakehouse --extras observability

# Or with pip
python -m pip install "fraudtwin[ml,graph]"
python -m pip install "fraudtwin[kafka,postgres,lakehouse,observability]"

The optional Spark reference pipeline is documented in docs/spark-streaming.md and lives under examples/spark-streaming/. It consumes either the observable Kafka payment-event contract or persisted PaymentEvent Parquet rows. Spark is not required for local generation.

For laptop validation, use only configs/scale-dev.yaml (1,000 target payments). Larger scale profiles are hardware-dependent benchmark targets and are not validated by CI or documentation examples.

Docker-backed examples are documented separately and always include an offline fallback. Logical Kafka chaos simulates message delivery semantics; it does not claim to reproduce physical packet loss or broker failures.

Design principles

• Determinism first. Named random streams isolate entities, behavior, payments, fraud, quality, graphs, and benchmarks.
• Truth is not availability. Oracle truth is never treated as an operational feature.
• Invariants are executable. Ledger balance, temporal cutoffs, identity, contracts, and reconciliation are validated in code.
• Integrations stay at the boundary. Kafka, databases, lakehouses, and serving adapters consume stable records rather than changing simulation semantics.
• Evidence over anecdotes. Every experiment can emit compact metrics, manifests, and fingerprints suitable for review or regression tests.

Contributing

Bug reports, improvements, documentation updates, and new integration tests are welcome. See DEVELOPMENT.md for the contributor setup, quality gate, focused tests, and contribution workflow. Pull-request CI runs the same checks automatically.

License

FraudTwin is released under the Apache License 2.0. See docs/references.md for the standards, repositories, and research that informed the framework.