synthetic-data-gen

Generate synthetic training data for ML pipelines — Q&A pairs, classification examples, tabular data, and instruction-following datasets.

Install

pip install synthetic-data-gen

Requires ANTHROPIC_API_KEY environment variable.

Quick start

from synth_data import SynthDataGen

gen = SynthDataGen()

# Q&A pairs from your corpus
qa = gen.qa_pairs(context="The UK AI Safety Institute was founded in 2023...", n=10)
qa.save("qa_train.jsonl")

# Classification examples
examples = gen.classification(
    labels=["compliant", "non_compliant", "requires_review"],
    domain="UK GDPR data processing records",
    n=60,
)
examples.save("gdpr_train.csv", format="csv")

# Instruction-following dataset
dataset = gen.instructions(
    task_description="Summarise UK government policy documents",
    n=30,
)
print(dataset.to_alpaca())  # Alpaca fine-tuning format

# Tabular synthetic data
employees = gen.tabular(
    columns=["name", "department", "grade", "salary"],
    schema={"grade": "one of: EO, HEO, SEO, G7, G6", "salary": "integer 25000-120000"},
    domain="UK civil service",
    n=100,
)
employees.save("workforce.csv", format="csv")

Export formats

dataset.to_json()    # pretty-printed JSON
dataset.to_jsonl()   # one object per line (HuggingFace format)
dataset.to_csv()     # CSV with headers
dataset.to_alpaca()  # Alpaca instruction-tuning format
dataset.save("file.jsonl", format="jsonl")

Linda Oraegbunam | LinkedIn | GitHub