data4ai 0.1.2

Production-ready AI-powered dataset generation for instruction tuning and model fine-tuning

Data4AI 🚀

AI-powered dataset generation for instruction tuning and model fine-tuning

Generate high-quality synthetic datasets using state-of-the-art language models through OpenRouter API. Perfect for creating training data for LLM fine-tuning.

✨ Key Features

• 🤖 100+ AI Models - Access to GPT-4, Claude, Llama, and more via OpenRouter
• 📊 Multiple Formats - Support for Alpaca, Dolly, ShareGPT schemas
• 🔮 DSPy Integration - Dynamic prompt optimization for better quality
• 💾 Excel/CSV Support - Start from templates or existing data
• ☁️ HuggingFace Hub - Direct dataset publishing
• ⚡ Production Ready - Rate limiting, checkpointing, deduplication

🚀 Quick Start

Installation

pip install data4ai              # Core features
pip install data4ai[excel]       # With Excel support
pip install data4ai[all]         # All features

Get API Key

Get your free API key from OpenRouter

export OPENROUTER_API_KEY="your_key_here"

Generate Your First Dataset

# Generate from description
data4ai prompt \
  --repo my-dataset \
  --description "Create 10 Python programming questions with answers" \
  --count 10

# View results
cat my-dataset/data.jsonl

📚 Common Use Cases

1. Generate from Natural Language

data4ai prompt \
  --repo customer-support \
  --description "Create customer support Q&A for a SaaS product" \
  --count 100

2. Complete Partial Data from Excel

# Create template
data4ai create-sample template.xlsx

# Fill some examples in Excel, leave others blank
# Then generate completions
data4ai run template.xlsx --repo my-dataset --max-rows 100

3. Publish to HuggingFace

# Generate and publish
data4ai prompt \
  --repo my-public-dataset \
  --description "Educational content about machine learning" \
  --count 200 \
  --huggingface

🐍 Python API

from data4ai import generate_from_description

result = generate_from_description(
    description="Create Python interview questions",
    repo="python-interviews",
    count=50
)

print(f"Generated {result.row_count} examples")

📋 Supported Schemas

Alpaca (Default - Instruction tuning)

{
  "instruction": "What is machine learning?",
  "input": "Explain in simple terms",
  "output": "Machine learning is..."
}

Dolly (Context-based)

{
  "instruction": "Summarize this text",
  "context": "Long text here...",
  "response": "Summary..."
}

ShareGPT (Conversations)

{
  "conversations": [
    {"from": "human", "value": "Hello"},
    {"from": "gpt", "value": "Hi there!"}
  ]
}

⚙️ Configuration

Create .env file:

OPENROUTER_API_KEY=your_key_here
OPENROUTER_MODEL=meta-llama/llama-3-8b-instruct  # Optional
HF_TOKEN=your_huggingface_token                   # For publishing

Or use CLI:

data4ai config --save

📖 Documentation

• Detailed Usage Guide - Complete CLI reference
• Examples - Code examples and recipes
• API Documentation - Python API reference
• Publishing Guide - PyPI publishing instructions
• All Documentation - Complete documentation index

🛠️ Development

# Clone repository
git clone https://github.com/zysec/data4ai.git
cd data4ai

# Install for development
pip install -e ".[dev]"

# Run tests
pytest

# Check code quality
ruff check .
black --check .

🤝 Contributing

Contributions welcome! Please check our Contributing Guide.

📄 License

MIT License - see LICENSE file.

🔗 Links

• PyPI Package
• GitHub Repository
• Documentation
• Issue Tracker

---

Made with ❤️ by ZySec AI