Synthetically Engineered Evaluation Data (SEED)

AI-powered synthetic data generation pipeline built on the Strands Agents SDK. Produces realistic PDF documents from JSON schemas, validates them through multi-stage critique loops, and optionally applies image augmentation to simulate real-world scanning/faxing artifacts. Each document is paired with a ground-truth JSON label, so the output is a ready-made benchmark set. SEED also generates structured (tabular) data — CSV, Parquet, Excel, or JSON, one file per entity — from the same planned schema.

Designed for building evaluation datasets for document understanding systems (OCR, Key Information Extraction (KIE), document classification) and for tabular datasets to exercise data pipelines, analytics, and ML workflows. Use it from the command line (seed-data) or the typed Python API (seed_data.Generator).

Not a production-ready solution. This asset represents a proof-of-value for the services included and is not intended as a production-ready solution. You must determine how the AWS Shared Responsibility Model applies to your specific use case and implement the controls needed to achieve your desired security outcomes. AWS offers a broad set of security tools and configurations to enable our customers.

Ultimately it is your responsibility as the developer of a full-stack application to ensure all of its aspects are secure. We provide security best practices in the repository documentation and a secure baseline, but Amazon holds no responsibility for the security of applications built from this tool.

⚡ New here? Read the Quickstart — install from PyPI and generate your first document in a few minutes.

Quick Start

Install from PyPI (the default renderer is pure Python — nothing else to set up):

pip install seed-data

# Configure AWS credentials with Bedrock access
export AWS_PROFILE=your-profile-name

# Copy the built-in schema library into a local, editable folder
seed-data clone-schema-library ./schemas

# Generate a single document (PDF + ground-truth JSON) into ./output
seed-data --schema-dir fcc-invoice --output ./output

--schema-dir accepts a bundled schema name (like fcc-invoice) or a path to a schema directory, so the command above works with or without the clone step.

Browse the schema library on GitHub: awslabs/…/schemas. See the full Quickstart for batches, augmentation, and model selection.

# Install (creates the venv and installs from uv.lock)
uv sync

export AWS_PROFILE=your-profile-name

# Generate a single FCC invoice
uv run seed-data --schema-dir fcc-invoice

# Generate a diverse batch of 4 invoices with augmentation (needs the [augment] extra;
# `uv sync` installs it via the dev group)
uv run seed-data --schema-dir fcc-invoice \
  --count 4 --augment \
  --scenario "FCC broadcast invoices for packaged food companies in the midwest"

Two ways to use it

Everything is available from both the command line and Python; they run the same pipeline and produce the same artifacts.

Command line:

seed-data --schema-dir invoice --scenario "Midwest food-distributor invoice"   # single
seed-data --schema-dir fcc-invoice --count 10 --scenario "Local TV stations"    # batch
seed-data packet lending-package --scenario "First-time homebuyer in Portland"  # packet
seed-data infer-schema ./samples/*.pdf --name invoice --output ./schemas/invoice  # schema from real docs
seed-data plan "Customers and their orders" --output ./schema.json              # any input -> schema
seed-data generate-structured ./schema.json --rows 500 --format parquet           # structured (tabular)
seed-data plan-and-generate "Customers and their orders" --output structured --rows 500         # plan + generate

Python:

from seed_data import Generator, ModelConfig

gen = Generator(models=ModelConfig(doc="gpt-oss", critic="haiku"), threshold=5)

doc    = gen.generate("invoice", scenario="Midwest food-distributor invoice")
batch  = gen.generate_batch("fcc-invoice", count=10, scenario="Local TV stations")
packet = gen.generate_packet("lending-package", scenario="First-time homebuyer in Portland")
schema = gen.infer_schema("./samples/*.pdf", name="invoice")  # reverse-engineer a schema from real docs
ing    = gen.plan("Customers and their orders")             # any input -> InferredSchema
table  = gen.generate_structured(ing, rows=500)               # structured (tabular) data
result = gen.plan_and_generate("Customers and their orders")                # plan + generate, one call

Schema from documents — the inverse of generation. Point SEED at real example documents (PDF/PNG/JPEG, local or s3://) and a vision model reverse-engineers a schema you can generate unlimited synthetic look-alikes from. Works for a single document type, or splits one concatenated multi-document PDF into a per-type packet. See Schema from Documents.

Full docs: CLI Usage · Python API Usage.

Structured Data Generation

Alongside documents, SEED generates structured (tabular) datasets. A run writes one file per entity into the output directory, named from the lowercased entity name with spaces replaced by underscores: <entity>.csv, .json, .xlsx, or .parquet. A schema with entities Customer and Order at --format csv produces output/customer.csv and output/order.csv.

Structured generation is an opt-in extra — pandas and the file-format engines (openpyxl for .xlsx, pyarrow for .parquet) are kept out of the base install so pip install seed-data stays lean for document-only users:

pip install "seed-data[structured]"

The document pipeline never needs the extra. Structured commands raise a clear ImportError pointing at it if it is missing. Every output format works with the extra installed — no separate engine install.

Planning is the shared front door. seed-data plan takes free text, example data files (CSV/JSON/XLSX), formal schemas (JSON Schema, SQL DDL), documents (PDF/PNG/JPEG, read with a vision model), ERD diagrams — or several of those together — and writes one unified InferredSchema JSON. The same planned schema can drive either modality: pass it to generate-structured for tables, or to generate-documents for PDFs.

# 1. Plan a schema from anything
seed-data plan "Customers with orders and line items" ./samples/orders.csv \
  --name retail --output ./schema.json

# 2. Generate tables from that schema
seed-data generate-structured ./schema.json --rows 500 --format csv --output ./output

# ...or generate documents from the very same schema
seed-data generate-documents ./schema.json --entity Order --count 3

seed-data plan-and-generate does planning + generation in one shot, with no intermediate schema file. Note that on plan-and-generate — and only on plan-and-generate — --output selects the modality and --output-dir selects the path:

seed-data plan-and-generate "Customers with orders and line items" ./samples/orders.csv \
  --output structured --rows 500 --format parquet \
  --output-dir ./output --save-schema ./schema.json

From Python, the same two steps with Generator:

from seed_data import Generator

gen = Generator(output_dir="./output")

schema = gen.plan("Customers with orders and line items", "./samples/orders.csv",
                    name="retail")
result = gen.generate_structured(schema, rows=500, format="csv")

print(result.success)       # True when files were written
print(result.output_paths)  # ['./output/customer.csv', './output/order.csv', ...]
print(result.format)        # 'csv'
print(result.row_counts)    # {'Customer': 500, 'Order': 500}
print(result.evaluation)    # metric name -> score
print(result.token_usage)   # {'inputTokens': ..., 'outputTokens': ..., 'totalTokens': ...}
print(result.error)         # None on success

Generator.available_input_types() lists what planning can classify: free_text, example_data, schema, document, erd.

A structured run writes a flat directory of per-entity files:

output/
├── customer.csv             # one file per entity, lowercased, spaces -> _
├── order.csv
└── order_line_item.csv

Architecture

Shared front door, two modalities

   free text ─┐
example data ─┤
      schema ─┼─→  plan  ─→ InferredSchema ─┬─→ structured pipeline ─→ CSV/Parquet/
   documents ─┤                             │                          Excel/JSON
         ERD ─┘                             └─→ document pipeline ─→ PDF + JSON label

plan classifies each input and normalizes everything into one InferredSchema, so the choice of modality is made after planning, not before it. One schema can therefore drive tables, documents, or both. The document pipeline is unchanged — --schema-dir with a legacy schema directory still enters it directly, without planning.

Single-document pipeline

data_generator → data_critic → [doc_generator → doc_critic loop] → (augmentor → aug_critic)
                 ↑ (reject)                      ↑ (reject)              ↑ (reject)
                 └─────────┘                     └──────────┘            └──────────┘

The data generator invents schema-conforming JSON; the data critic validates it (deterministic JSON-Schema check + LLM domain review); the doc generator writes HTML/CSS and renders a PDF; the vision doc critic reviews the render and loops until it passes the threshold. With --augment, an augmentor applies aging effects and an aug critic checks legibility.

Batch (concurrent fan-out)

A batch turns one high-level --scenario into N distinct scenarios, then runs a self-contained pipeline graph for each as sibling nodes — the Strands graph executes them concurrently (no thread pools or worker flags). A specific scenario produces far more varied output than a generic one:

# Generic — less diverse
--scenario "Generate diverse FCC invoices"

# Specific — much more diverse
--scenario "FCC broadcast invoices for packaged food companies advertising in the midwest"
--scenario "Local car dealership TV ads across small-market stations in the southeast"

Packet (coordinated multi-document)

A packet is a set of different document types that share context — like a loan application with a credit report, pay stubs, and bank statements, all for the same applicant — merged into one multi-page PDF. See the Packets guide.

Agent Descriptions

Setup

This project uses uv and ships a committed uv.lock for reproducible installs. uv is the recommended workflow.

# Install uv (see https://docs.astral.sh/uv/getting-started/installation/)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create the virtual environment and install all dependencies from uv.lock,
# including the dev group (pytest, ruff, docs). Requires Python 3.12+.
uv sync

# Run any command inside the managed environment with `uv run`, e.g.
uv run seed-data --help

uv sync installs the dev dependency group by default (configured via [tool.uv] default-groups). Use uv sync --no-dev for a runtime-only install. The dev group already includes the structured stack, so a uv sync checkout can generate both modalities with no extra step.

# pip (add the [dev] extra for pytest/ruff/docs tooling)
pip install -e ".[dev]"

# or conda for the environment, then pip for the package
conda create -n seed python=3.12 -y
conda activate seed
pip install -e ".[dev]"

Optional dependencies (extras)

The base install is CI-enforced to import and pass its test suite with no pandas present, so the published document-only offering keeps working exactly as before. Structured commands raise a clear ImportError telling you to install the extra; the document pipeline never needs it. All four output formats — including --format parquet — work with [structured] alone.

Augmentation is opt-in too: --augment (or augment=True) without the [augment] extra fails immediately with an install hint, before any model call. Keeping opencv out of the base install is what lets it build on Alpine/musl images, where opencv ships no wheels.

PDF renderers

The default renderer is xhtml2pdf (pure Python, no system libraries) — a fresh pip install seed-data renders PDFs out of the box. Select a renderer with --renderer:

WeasyPrint only — install its system libraries first:

# macOS
brew install pango gdk-pixbuf libffi

# Ubuntu/Debian
apt-get install libpango-1.0-0 libgdk-pixbuf2.0-0

Usage

Single document

seed-data --schema-dir fcc-invoice \
  --scenario "Local TV station in Portland, Oregon"

Single document with augmentation

seed-data --schema-dir fcc-invoice \
  --scenario "Local TV station in Portland, Oregon" \
  --augment

Batch generation

--count > 1 switches into batch mode and fans out concurrent pipeline graphs:

seed-data --schema-dir fcc-invoice \
  --count 10 \
  --scenario "CPG brands on local TV stations in the American southwest" \
  --augment

Packet generation

seed-data packet lending-package \
  --count 3 --doc-workers 2 --augment \
  --scenario "First-time homebuyers in the Pacific Northwest, 30-year fixed mortgage"

CLI Options

Common flags (single-document and batch):

Packet subcommand (seed-data packet <name|path>):

Plan subcommand (seed-data plan <inputs...>) — any inputs to one InferredSchema JSON:

Structured subcommand (seed-data generate-structured <schema>):

Document subcommand (seed-data generate-documents <schema>) — the modern spelling of the default mode, and what lets a planned schema drive the document pipeline:

End-to-end subcommand (seed-data plan-and-generate <inputs...>) — plan, then generate, in one shot with no intermediate schema file.

On plan-and-generate, --output is the modality, not a path. The path is --output-dir. Every other command uses --output for the path.

Utility subcommand — copy the bundled schema library out to edit locally:

seed-data clone-schema-library ./schemas

Run seed-data --help, seed-data packet --help, seed-data infer-schema --help, seed-data plan --help, seed-data generate-structured --help, seed-data generate-documents --help, and seed-data plan-and-generate --help for the complete list.

What to expect from a batch

• A single document takes roughly 30–100 seconds end-to-end.
• All workers launch concurrently. The Python Strands graph has no built-in concurrency cap, so large counts may hit Bedrock rate limits (adaptive retries absorb this, but wall-clock grows). Start with modest counts.
• Partial failures are normal. Failed documents come back with success=False and verdict="error"; the rest still complete.
• Use a specific --scenario. Vague scenarios produce same-y output.

Available Models

seed-data --help lists every available model key.

Schema Directories

Each document type lives in its own directory:

schemas/fcc-invoice/
├── schema.json              # JSON Schema — REQUIRED
├── generation_guidance.md   # Visual style, data ranges, math rules — optional
└── samples/                 # Reference PDFs for the doc critic — optional, local-only
    ├── README.md            # What samples are for + handling rules
    └── *.pdf                # git-ignored — never committed

samples/ (reference documents) — important

The optional samples/ directory holds real example PDFs used only by the document critic as a visual style reference — they are never seen by the generation stages and no content from them is reproduced in the output. See critique.py and any schemas/<type>/samples/README.md for the full explanation.

Sample PDFs are local-only and git-ignored. Do not commit them: real documents may contain third-party names or PII you don't have the right to redistribute, which can trigger data-distribution policies and block open-source release. You are responsible for having the rights to any sample you add. Use --no-critic-samples to skip samples entirely. The repository ships with no sample documents.

generation_guidance.md

This is the key file for controlling document quality and diversity. It tells agents:

• Visual style: layout, fonts, density, colors
• Data realism: value ranges, naming conventions, line item counts
• Math rules: arithmetic constraints (e.g., "GrossTotal = sum of LineItemRate")
• Common issues to avoid: known pitfalls for this document type
• Generation Choices: optional fields with x-probability and table presentation variations (see below)

The scenario planner also reads this file to create diverse scenarios that respect the document type's constraints.

Generation Choices

Document diversity comes from two sources of per-run variation:

1. Optional fields with x-probability — schema fields marked with an x-probability annotation are independently included or omitted per document based on their probability.
2. Table presentation variations — sections that can render either as tables or inline fields, declared in generation_guidance.md with ~50/50 probabilities.

See docs/docs/Guides/generation-choices.md for the full guide: conventions, examples for nested and array fields, and how to add it to a new document type.

Output Structure

Single-document and batch runs write:

output/
├── pdfs/                    # Clean PDFs
├── data/                    # Ground-truth JSON label per document
├── generation_scripts/      # HTML files used to render PDFs
├── augmented/               # Augmented PDFs (when --augment)
└── config/                  # Copy of schema + guidance for reproducibility

Packet runs use the evaluation-dataset layout (merged PDFs in input/, per-section labels in baseline/); see the Packets guide.

Structured runs (generate-structured, or plan-and-generate --output structured) write one file per entity directly into the output directory — no subdirectories:

output/
├── customer.csv             # lowercased entity name, spaces -> underscores
├── order.csv
└── order_line_item.csv

The extension follows --format: .csv, .json, .xlsx (excel), or .parquet. StructuredResult.output_paths lists the files that were written.

Tests

# Run the full unit-test suite (no LLM calls). Integration tests are excluded
# by default (see addopts in pyproject.toml).
uv run pytest

# Or run a single module
uv run pytest tests/test_unit.py -v

# CLI smoke tests (no Bedrock) — verify the command works out of the box
uv run pytest tests/test_cli_smoke.py -v

Packet Configuration

Packet configs live in src/seed_data/packets/<packet-name>/packet.json. See the Packets guide for the full guide including config fields, shared context modes, label format, and architecture.

Packet config example

{
  "name": "insurance-claim-packet",
  "description": "Property insurance claim submission packet.",
  "documents": [
    {
      "document_class": "Insurance Claim",
      "schema_dir": "insurance-claim",
      "required": true,
      "min_instances": 1,
      "max_instances": 1
    },
    {
      "document_class": "Invoice",
      "schema_dir": "invoice",
      "required": true,
      "min_instances": 1,
      "max_instances": 3
    }
  ],
  "shared_context": {
    "claimant_name": "Full legal name of the person filing the claim",
    "claimant_address": "Street address of the claimant"
  },
  "shared_context_hint": "All documents relate to the same insurance claim."
}

Security

seed-data runs AI agents that write and execute code and read/write files on the host in order to render documents. Read this section before running it on anything you don't fully trust. See also the disclaimer at the top of this README — you are responsible for the security of anything you build with this tool.

Data provenance

Code & file execution (most important)

The document-generation agent uses tools that act on the host with the privileges of the user running the CLI:

• Code execution — with the reportlab renderer the agent runs Python via a shell tool; with the HTML renderers (xhtml2pdf/weasyprint) it renders agent-authored HTML. Treat all agent-generated code in output/generation_scripts/ as untrusted model output — do not blindly re-run it later.
• Filesystem access — file read/write tools operate on model-supplied paths with no built-in sandbox or path allowlist.
• Tool consent — the Strands Agents SDK can prompt for human approval before executing tools. Setting BYPASS_TOOL_CONSENT=true disables that prompt for autonomous runs; leave it unset when running on untrusted input so you can approve tool calls.

Recommendation: run the pipeline in an isolated environment (container or VM) with no sensitive data on disk, restricted network egress, and dedicated, least-privilege AWS credentials — not your personal developer profile.

Untrusted schemas, guidance, and scenarios

schema.json, generation_guidance.md, sample PDFs, and the --scenario text are all injected into agent prompts. Treat content from untrusted sources as a prompt-injection vector: a malicious schema or guidance file could attempt to steer an agent into misusing the code/file tools. Only run schemas you trust, or run inside the isolated environment described above.

AWS credentials

Model calls go to Amazon Bedrock using the credentials resolved from your AWS_PROFILE (or an explicit boto3 session passed to Generator). Any code the agent executes inherits that environment, so those credentials — and whatever they can access — are reachable by agent-run code. Use a scoped, Bedrock-only profile and avoid broad/admin credentials.

Cost

Generation runs make repeated Bedrock calls and use critic-driven retry loops. Large batch counts or high --max-attempts increase token spend. Keep --max-attempts/--timeout conservative and monitor Bedrock usage.