synthetic-dataset-sdk 0.1.2

A lightweight Python client for the Synthetic Dataset Generation SaaS API

Synthetic Dataset SDK

A lightweight Python client for the Synthetic Dataset Generation API. This SDK allows you to easily ingest document pages and retrieve generated synthetic datasets.

Installation

pip install synthetic-dataset-sdk

Quick Start

from synthetic_dataset_sdk import SyntheticDatasetClient

# 1. Initialize the client
# To get an sdg_key, first create a project via the REST API (POST /projects/)
sdk = SyntheticDatasetClient(sdg_key="your-sdg-key")

# 2. Upload document pages
# pages should be a list of dicts: [{"page_no": 1, "doc_id": "DOC-A", "text": "..."}]
pages = [
    {
        "page_no": 1, 
        "doc_id": "DOC-A", 
        "text": "The quick brown fox jumps over the lazy dog."
    }
]
result = sdk.upload(pages)
print(f"Project ID: {result['project_id']}")
print(f"Total batches: {result['total_batches']}")

# 3. Check batch formation status
status = sdk.get_batch_status()
print(f"Processed: {len([b for b in status['batches'] if b['status'] == 'done'])} / {status['total_batches']}")

# 4. Fetch the generated dataset (paginated)
# This will return validated QA pairs once the pipeline is done
dataset = sdk.get_dataset(include_faulty=False, page=1, page_size=20)
for entry in dataset["entries"]:
    print(f"Q: {entry['question']}")
    print(f"A: {entry['answer']}")

Features

• Easy Ingestion: Upload raw text pages directly from your pipeline.
• Batch Tracking: Monitor the progress of batch formation and QA generation.
• Faulty Filtering: Toggle between high-accuracy pairs and the full generated set.
• Lightweight: Minimal dependencies (just requests).

Requirements

• Python 3.8+
• requests