SANTIQ 0.1.2

A lightweight, modular, plugin-first ETL platform

Santiq

A lightweight, modular, plugin-first ETL platform designed for individuals, small businesses, and scalable up to enterprise workloads.

Quick Start • Documentation • Examples • Contributing

---

🚀 What is Santiq?

Santiq is a modern ETL (Extract, Transform, Load) platform that makes data processing simple, reliable, and extensible. Built with a plugin-first architecture, Santiq allows you to:

• Extract data from any source (files, databases, APIs, cloud services)
• Profile data to automatically detect quality issues
• Transform data with intelligent cleaning and validation
• Load data to any destination with full audit trails

✨ Key Features

• 🔌 Plugin-First Architecture: Everything is a plugin, even core functionality
• 🧠 Smart Data Profiling: Automatic issue detection with context-aware fix suggestions
• ⚡ Multiple Execution Modes: Manual, half-automatic, and controlled-automatic
• 📊 Learning System: Remembers your preferences for future pipeline runs
• 🛡️ Enterprise Ready: Comprehensive audit logging and error handling
• 🚀 Performance Optimized: Hybrid memory/disk usage based on data size
• 🔧 Extensible: Easy to create custom plugins for any data source or transformation

🏃‍♂️ Quick Start

Installation

pip install santiq

Your First Pipeline

# Initialize a new pipeline
santiq init my-first-pipeline

# Edit the generated configuration file
# my-first-pipeline.yml

# Run the pipeline
santiq run pipeline my-first-pipeline.yml

Example Pipeline Configuration

name: "Customer Data Cleaning"
description: "Clean and validate customer data from CSV"

extractor:
  plugin: csv_extractor
  params:
    path: "${INPUT_PATH}/customers.csv"
    header: 0

profilers:
  - plugin: basic_profiler
    params: {}

transformers:
  - plugin: basic_cleaner
    params:
      drop_nulls: true
      drop_duplicates: true
      convert_types:
        age: numeric
        signup_date: datetime

loaders:
  - plugin: csv_loader
    params:
      path: "${OUTPUT_PATH}/cleaned_customers.csv"

📚 Documentation

For Users

• Getting Started Guide - Complete beginner's guide
• User Guide - Comprehensive usage instructions
• Configuration Reference - Pipeline configuration options
• CLI Reference - Command-line interface documentation

For Developers

• Plugin Development Guide - Create custom plugins
• API Reference - Core API documentation
• Plugin Examples - Sample plugin implementations

For Administrators

• Installation Guide - Production deployment
• Configuration Management - Environment setup
• Monitoring & Logging - Audit trails and observability

🔌 Plugin Ecosystem

Santiq comes with built-in plugins and supports a growing ecosystem:

Built-in Plugins

• Extractors: CSV, JSON, Excel files
• Profilers: Basic data quality analysis
• Transformers: Data cleaning, validation, type conversion
• Loaders: CSV, JSON, database outputs

Community Plugins

# Install community plugins
santiq plugin install santiq-plugin-postgres
santiq plugin install santiq-plugin-elasticsearch

# List available plugins
santiq plugin list --available

🎯 Use Cases

Data Quality Assurance

# Profile data and get quality report
santiq run pipeline data-quality.yml --mode manual

Automated Data Processing

# Run with automatic fix application
santiq run pipeline production.yml --mode controlled-auto

Data Migration

# Migrate data between systems
santiq run pipeline migration.yml --mode half-auto

🏗️ Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Extractors    │    │    Profilers     │    │  Transformers   │
│  ┌───────────┐  │    │  ┌────────────┐  │    │  ┌───────────┐  │
│  │    CSV    │  │    │  │   Basic    │  │    │  │  Cleaner  │  │
│  │  Database │  │────┤  │  Advanced  │  ├────┤  │   AI Fix  │  │
│  │    API    │  │    │  │   Schema   │  │    │  │  Custom   │  │
│  └───────────┘  │    │  └────────────┘  │    │  └───────────┘  │
└─────────────────┘    └──────────────────┘    └─────────────────┘
         │                        │                        │
         └────────────────────────┼────────────────────────┘
                                  │
                    ┌─────────────────────────┐
                    │     Santiq Engine       │
                    │  ┌─────────────────┐    │
                    │  │ Plugin Manager  │    │
                    │  │ Audit Logger    │    │
                    │  │ Config Manager  │    │
                    │  └─────────────────┘    │
                    └─────────────────────────┘
                                  │
                    ┌─────────────────────────┐
                    │       Loaders           │
                    │  ┌─────────────────┐    │
                    │  │      CSV        │    │
                    │  │    Database     │    │
                    │  │   Cloud Storage │    │
                    │  └─────────────────┘    │
                    └─────────────────────────┘

🛠️ Development

Setting Up Development Environment

git clone https://github.com/yourusername/santiq.git
cd santiq
pip install -e ".[dev]"
pre-commit install

Running Tests

pytest
pytest --cov=santiq tests/  # With coverage

Creating a Plugin

See Plugin Development Guide for detailed instructions.

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Ways to Contribute

• 🐛 Report Bugs: Create detailed issue reports
• 💡 Suggest Features: Propose new functionality
• 🔧 Submit Code: Fix bugs or add features
• 📚 Improve Docs: Help make documentation better
• 🔌 Create Plugins: Build plugins for new data sources

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built with Pandas for data manipulation
• Powered by Typer for CLI
• Styled with Rich for beautiful output
• Validated with Pydantic for data validation

---

Made with ❤️ by the Santiq Community

GitHub • Issues • Discussions