sdg-hub 0.2.0

Synthetic Data Generation

sdg_hub: Synthetic Data Generation Toolkit

A modular Python framework for building synthetic data generation pipelines using composable blocks and flows. Transform datasets through building-block composition - mix and match LLM-powered and traditional processing blocks to create sophisticated data generation workflows.

📖 Full documentation available at: https://ai-innovation.team/sdg_hub

✨ Key Features

🔧 Modular Composability - Mix and match blocks like Lego pieces. Build simple transformations or complex multi-stage pipelines with YAML-configured flows.

⚡ Async Performance - High-throughput LLM processing with built-in error handling.

🛡️ Built-in Validation - Pydantic-based type safety ensures your configurations and data are correct before execution.

🔍 Auto-Discovery - Automatic block and flow registration. No manual imports or complex setup.

📊 Rich Monitoring - Detailed logging with progress bars and execution summaries.

🧩 Easily Extensible - Create custom blocks with simple inheritance. Rich logging and monitoring built-in.

📦 Installation

Recommended: Install uv — see https://docs.astral.sh/uv/getting-started/installation/

# Production
uv pip install sdg-hub

# Development
git clone https://github.com/Red-Hat-AI-Innovation-Team/sdg_hub.git
cd sdg_hub
uv pip install .[dev]
# or: uv sync --extra dev

Optional Dependencies

# For vLLM support
uv pip install sdg-hub[vllm]

# For examples
uv pip install sdg-hub[examples]

🚀 Quick Start

🧱 Core Concepts

Blocks are composable units that transform datasets - think of them as data processing Lego pieces. Each block performs a specific task: LLM chat, text parsing, evaluation, or transformation.

Flows orchestrate multiple blocks into complete pipelines defined in YAML. Chain blocks together to create complex data generation workflows with validation and parameter management.

# Simple concept: Blocks transform data, Flows chain blocks together
dataset → Block₁ → Block₂ → Block₃ → enriched_dataset

Try it out!

Flow Discovery

from sdg_hub import FlowRegistry

# Auto-discover all available flows (no setup needed!)
FlowRegistry.discover_flows()

# List available flows
flows = FlowRegistry.list_flows()
print(f"Available flows: {flows}")

# Search for specific types
qa_flows = FlowRegistry.search_flows(tag="question-generation")
print(f"QA flows: {qa_flows}")

Using Flows

from sdg_hub import FlowRegistry, Flow
from datasets import Dataset

# Load the flow by name
flow_name = "Advanced Document Grounded Question-Answer Generation Flow for Knowledge Tuning"
flow_path = FlowRegistry.get_flow_path(flow_name)
flow = Flow.from_yaml(flow_path)

# Discover recommended models
default_model = flow.get_default_model()
recommendations = flow.get_model_recommendations()

# Configure model settings at runtime
# This assumes you have a hosted vLLM instance of meta-llama/Llama-3.3-70B-Instruct running at http://localhost:8000/v1
flow.set_model_config(
    model=f"hosted_vllm/{default_model}",
    api_base="http://localhost:8000/v1",
    api_key="your_key",
)

# Create your dataset with required columns
dataset = Dataset.from_dict({
    'document': ['Your document text here...'],
    'document_outline': ['1. Topic A; 2. Topic B; 3. Topic C'],
    'domain': ['Computer Science'],
    'icl_document': ['Example document for in-context learning...'],
    'icl_query_1': ['Example question 1?'],
    'icl_response_1': ['Example answer 1'],
    'icl_query_2': ['Example question 2?'], 
    'icl_response_2': ['Example answer 2'],
    'icl_query_3': ['Example question 3?'],
    'icl_response_3': ['Example answer 3']
})

# Generate high-quality QA pairs
result = flow.generate(dataset)

# Access generated content
questions = result['question']
answers = result['response']
faithfulness_scores = result['faithfulness_judgment']
relevancy_scores = result['relevancy_score']

Quick Testing with Dry Run

# Test the flow with a small sample first
dry_result = flow.dry_run(dataset, sample_size=1)
print(f"Dry run completed in {dry_result['execution_time_seconds']:.2f}s")
print(f"Output columns: {dry_result['final_dataset']['columns']}")

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines on how to contribute to this project.

---

Built with ❤️ by the Red Hat AI Innovation Team