Most synthetic data tools fit a model to data you already have. That is no help when you have none, cannot get access, or are not allowed to put what you have into a model.

synthweave generates from a schema you declare by hand, then optionally layers in statistical structure, realistic messiness, and identifiers that link correctly across tables. It works the other way round too:

New here? Every term below is defined in the full guide: glossary, complete syntax reference, and an 8-step tutorial. This README is the fast tour.

Install

pip install synthweave

pip install "git+https://github.com/emmanuelwunjc/synthweave.git"

git clone https://github.com/emmanuelwunjc/synthweave.git
cd synthweave
pip install -e ".[dev]"

Quickstart

import synthweave as sw

people = sw.Entity("person", count=10_000, attributes={"education": ["HS", "College"]})
table  = sw.Table("roster", grain="person", carry="*")
result = sw.Pipeline(sw.Schema([people], [table])).run()

result["roster"].head()

Ten thousand rows. No data file, no API key, no setup beyond pip install.

That is the whole shape of synthweave: describe a population once (Entity), describe a table that records it (Table), run it (Pipeline). Everything below is a variation on those three lines.

Why synthweave

The gap it targets: you need data that looks and behaves like a real linked administrative dataset (a school roster, a wage file, a benefits record, all describing the same people), and you either cannot access real microdata or are only prototyping and do not want to touch it yet.

How it works

A pipeline is up to four stages, each optional past the first:

Three properties hold it together.

Nothing is random. Every value is a pure function of (seed, stable key, salt). Same seed, same output, forever, regardless of chunk size. That is what lets two tables generated years apart still link correctly with no lookup table involved.

Structure has to come from somewhere. A model fit on independently generated columns learns nothing, so the synthesizer always names a structure source:

• Declared learns from relationships your own Conditional rules already put in the data. Use when you have domain knowledge but no data.
• Empirical learns from a real sample you supply.
• Prior learns from published statistics. No rows needed.

Every config number says where it came from. Wrap a guess in sw.modeled(0.08, "not yet sourced") and a real figure in sw.cited(0.82, "NCES Table 219.10"). Then result.unjustified() lists every number nobody has defended yet, so a config is auditable instead of a wall of magic numbers.

There is also a reduced-typing shorthand for common cases (a bare list instead of sw.Choice([...]), grain="person" instead of sw.PerEntity("person")). See the fast path.

Three ways to use it

Pick the one matching what you actually have. All three share this setup:

people = sw.Entity("person", count=10_000, attributes={"education": ["HS", "College"]})
wages  = sw.Table("wages", grain="person", carry=["education"], columns={"wage": 0.0})
schema = sw.Schema([people], [wages])

1. Domain knowledge only

Declare the relationship yourself. Conditional puts real structure in before anything is fitted, so wage depends on education because you said so, not because a model guessed. No synthesizer= needed.

wages = sw.Table("wages", grain="person", carry=["education"], columns={
    "wage": sw.Conditional("education", {
        "HS":      sw.Normal(38_000, 9_000, low=0),
        "College": sw.Normal(64_000, 18_000, low=0),
    }),
})
result = sw.Pipeline(sw.Schema([people], [wages])).run()

2. A real sample you want more of

Keep the 0.0 placeholder (the synthesizer overwrites it) and hand over your real data. It learns the actual relationships and generates rows that carry them.

real_sample = pd.read_csv("my_real_wages.csv")   # columns: education, wage

result = sw.Pipeline(
    schema,
    synthesizer=sw.CARTSynthesizer(["wage"], predictors=["education"], structure=real_sample),
).run()

3. Published statistics, or nothing at all

A report says "60% HS, 40% College"? Use it directly, no rows needed:

result = sw.Pipeline(
    schema,
    synthesizer=sw.CARTSynthesizer(["education"], structure={"education": {"HS": 0.6, "College": 0.4}}),
).run()

If you do not even have that, borrow real government survey data as a stand-in:

from synthweave.connectors.acs_pums import fetch_pums

acs = fetch_pums(["AGEP", "PINCP"], state="NY").rename(columns={"AGEP": "age", "PINCP": "income"})
synthesizer = sw.CARTSynthesizer(["age", "income"], structure=acs)

Worth knowing: this is real data, but not your population. It is a Census survey sample for a given state and year, so treat it as a plausible reference distribution, not a guarantee it matches your target demographics. Needs a free Census API key as CENSUS_API_KEY.

All three run end to end in examples/three_layers_data_availability.py.

FAQ

Scale

Stages pass chunks, not whole tables, so memory stays flat as row counts grow. Measured at roughly 200,000 rows/s with peak RSS around 0.44 GB, streamed straight to disk, on a 3.2 million row run.

The memory figure is the load-bearing one: it does not grow with row count, so tens of millions of rows is a matter of time, not RAM. Throughput varies with machine load, so treat it as an order of magnitude, not a benchmark.

sw.Pipeline(schema, chunk_size=200_000).run_to("out/", format="parquet")

Status

v0.2. Pre-1.0: the public API may still change between minor versions.

Every capability described above is present and tested. One implementation ships per stage, and the interfaces are the real commitment, so adding implementations should not require restructuring.

The API freezes at v1.0, which also requires zero known bugs in the "produces plausible-looking wrong data without raising" class. Those are tracked openly in the issue tracker along with the design reasoning behind them.

Not yet included: disclosure-risk metrics, differential privacy, R interop, deep generative synthesizers.

Development

pip install -e ".[dev]"
pre-commit install
PYTHONPATH=src python -m pytest tests/ -q

Tests run offline and go through the public API only, so internal stage code can be rewritten without touching them. Every bug fix is checked by reverting it and confirming a test goes red: a fix nothing catches is not a fix. See CONTRIBUTING.md.

Where this came from

The design generalizes two patterns proven in a real engagement building linked synthetic administrative data: deterministic hash-derived identifiers, and provenance-tagged config. It draws on pseudopeople for realistic noise models, and synthpop / tidysynthesis for sequential CART synthesis. What is new is combining them on a schema you define yourself, with no real data required to start.

Also runs in a browser. synthweave-frontend runs this pipeline locally, checks the linking guarantee against the produced data, and re-runs your config at three chunk sizes to prove the output is identical.

License

MIT