SynthWorld

Faker generates rows. SynthWorld generates connected identity worlds with adversarial evidence and an answer key.

SynthWorld creates deterministic, safely fictional populations for evaluating privacy, PII-extraction, entity-resolution, relationship-inference, agent-authority, and exposure-analysis systems. Selected benchmark families expose separately serialized product-safe observations; other artifacts are evaluator bundles that retain answer keys for scoring.

SynthWorld began as the ground-truth harness for Idcognito and is deliberately usable as an independent Apache-2.0 Python package. It is not an anonymisation tool and does not transform sensitive real-world data into a safe dataset.

Choose what you want to do

New to benchmark evaluation? The user guide explains the workflow in plain language, provides a five-minute walkthrough, and shows where your own system plugs into each current use case.

Why SynthWorld

A generated row can test whether a field accepts an email address. A SynthWorld benchmark can measure whether a system extracts that address from a document, links several conflicting records to the correct entity, infers only supported relationships, and assigns the expected exposure score.

Current benchmark families

• Core identity world: seeded personas, identity attributes, and evidence-backed relationships.
• Exposure corpus: breach, broker, search, and social observations, including zero-exposure controls, search collisions, and broker reappearance.
• Exact-span extraction: a product-safe public page corpus and a physically separate exact-span answer key, plus an annotated evaluator bundle that pairs the two for offline scoring.
• Entity resolution: opaque records and adversarial cases with separate entity-membership truth.
• Relationship inference: public association evidence, reciprocal positive cases, and unilateral negative controls.
• Risk calibration: provider-neutral breach observations with separately checksummed score, band, and factor truth.
• Asteria Agentic v1: ordered agent/runtime/delegation events, an oracle-free observed-action interface, and separate authority, attribution, temporal, and provenance truth.

The core-world, exposure-corpus, extraction-corpus, connection-benchmark, risk-benchmark, and agentic schemas are independently versioned 1.0.0 contracts. See DATA_DICTIONARY.md for field definitions and the strict public/oracle boundary. See GOLDEN_REVIEW.md for the frozen benchmark review record.

Public input and evaluator truth

Extraction, connection, risk, and Asteria Agentic each provide separately serialized product-safe input and physically separate evaluator truth. The first three use PublicExtractionCorpus, PublicConnectionCorpus, and PublicRiskCorpus; Asteria uses a multi-file public package. Extraction also ships an ExtractionCorpus annotated bundle, in which every AnnotatedExtractionPage embeds both the safe page and its answer_key, for offline evaluators; that bundle is convenient but is not a product-safe input.

The separated evaluation flow is:

product or model                    evaluator
       |                                |
       v                                v
public observations  ---------->  system predictions
                                          |
                                          v
                               separate answer key
                                          |
                                          v
                                  scored results

Only corpus types and CLI commands explicitly described as public should be passed to product adapters. Do not pass the annotated extraction corpus into a product or model without first projecting only its page fields.

Install

The distribution is published as idcognito-synthworld; the import package and the CLI are both named synthworld, and the package ships typed (py.typed). Release notes live in CHANGELOG.md.

SynthWorld requires Python 3.12 or newer; Python 3.11 and earlier are not supported.

pip install idcognito-synthworld
synthworld generate --seed 20260719 --persona-count 10 --output world.json

Selected frozen golden benchmarks are also browsable as tables on Hugging Face, byte-identical to the artifacts shipped in this package. The maintained dataset-card source and Asteria download instructions live in huggingface/README.md.

Develop from source

Install uv, clone the repository, and run:

uv sync --locked --all-groups
uv run synthworld generate --seed 20260719 --persona-count 10 --output world.json
uv run synthworld metrics --seed 20260719 --persona-count 10

Useful corpus commands include:

uv run synthworld generate-corpus --seed 20260719 --persona-count 10 --output exposures.json
uv run synthworld generate-public-extraction --seed 20260719 --persona-count 10 --output extraction.json
uv run synthworld generate-public-connections --seed 20260719 --persona-count 10 --output connections.json
uv run synthworld generate-risk-public --seed 20260719 --persona-count 10 --output risk.json
uv run synthworld generate-agentic --output asteria-agentic-v1

Use Asteria Agentic v1

Export the frozen world, then give only its public/ directory to the system under test:

synthworld generate-agentic --output asteria-agentic-v1
jq -c 'select(.payload.event_type == "action_attempted")' \
  asteria-agentic-v1/public/public_events.jsonl

Your adapter must write one ObservedActionTrace JSON object per action event. The repository's deliberately imperfect current-state baseline demonstrates the public-only integration path and writes a CLI-ready trace:

uv run python examples/evaluate_all.py --predictions-dir predictions
uv run synthworld evaluate agentic \
  --predictions predictions/agentic.jsonl \
  --summary

Replace current_state_agentic_trace in the example with your own policy, agent-observability, or audit system. Keep asteria-agentic-v1/evaluator/ out of that adapter; the SynthWorld scorer joins the answer key only after predictions have been produced. See the Asteria guide for the JSONL schema, Python API, replay rules, checksum verification, and metric definitions.

Custom agentic worlds built with build_agentic_benchmark are fully replayed and relationally validated before evaluator truth is created. Malformed runtime/agent, credential, delegator, actor, and owner-chain joins are rejected; truthful unauthorized attempts remain scoreable denials. Agentic scoring protocol 0.2.0 also distinguishes missing evidence from fabricated extras with completeness, exact-match, and micro-precision metrics.

See the user guide for goal-led walkthroughs, examples/ for runnable adapters and annotated sample output, and BENCHMARKS.md for reference baseline results and visual demonstrations.

The generate-extraction, generate-extraction-answers, generate-connection-benchmark, and generate-risk-answer commands include or emit evaluator-only truth. Keep those artifacts outside product and demo data paths. The generate-public-extraction, generate-public-connections, and generate-risk-public commands emit the separately serialized product-safe observations.

Evaluate a system

SynthWorld provides a unified command line tool to score predictions against separately serialized ground-truth answer keys:

synthworld evaluate <task> --predictions PATH [--seed S] [--persona-count N] [--summary]

Where task is one of agentic, extraction, entity-resolution, relationship, or risk. Agentic predictions use JSONL; the other tasks use JSON.

• --predictions: Path to the system predictions JSON or JSONL file, conforming to the task-specific schema.
• --seed: The benchmark seed used to load/generate matching ground-truth (ignored for frozen Asteria Agentic v1).
• --persona-count: The benchmark persona count (ignored for entity-resolution and Asteria Agentic v1).
• --summary: If provided, outputs a clean, compact terminal table summarizing the metrics instead of the raw JSON report.

Examples:

synthworld evaluate extraction --predictions predictions.json --seed 20260719 --summary
synthworld evaluate agentic --predictions observed-actions.jsonl --summary

Start with the user guide for runnable examples and score interpretation. See DATA_DICTIONARY.md for the full prediction and report schemas.

Roadmap and integrations

SynthWorld is intended to remain a focused ground-truth identity layer rather than become a second general-purpose simulator. Planned work is organised as packs and adapters:

• data-broker deletion and reappearance for Idcognito;
• broader AI-agent and non-human identity profiles for ZeroID, Arena, and EADS, building on the available Asteria Agentic v1 conformance fixture;
• enterprise IAM and identity-governance scenarios;
• LLM, RAG, and agent-memory privacy evaluation;
• digital-wallet and verifiable-credential testing;
• disaster identity continuity for Aftershock.

The phased plan, architecture boundaries, and tracking issues are documented in ROADMAP.md.

Verify every claim

make ci runs formatting, linting, strict type checking, all tests with 100% branch coverage, benchmark metrics at 10- and 100-persona scales, package inspection, and an isolated-wheel smoke test. The same gates run on Python 3.12 and 3.14 in GitHub Actions; a separate workflow job scans the repository's full history for secrets.

make ci

Generated JSON is safely fictional for fixtures, demos, tutorials, and evaluation when its synthetic markers remain intact. That safety property does not make every artifact oracle-free product input; use only explicitly public corpora for product adapters. SynthWorld is not a source of real identity data and must never be used to impersonate, target, or investigate a person. Do not replace the safeguards with plausible real-world identifiers.

License

Copyright 2026 Redoubt Labs ltd. Licensed under the Apache License 2.0.